[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:54:56 dist-test-slave-q5f0 krb5kdc[15440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:54:56 dist-test-slave-q5f0 krb5kdc[15440](info): set up 2 sockets
Sep 29 02:54:56 dist-test-slave-q5f0 krb5kdc[15440](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 02:54:57.877998 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:54:57 dist-test-slave-q5f0 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:54:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:54:58.044258 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:34989
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:58.367022 15453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:58.367337 15453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:58.367452 15453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:58.375306 15453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:54:58.375442 15453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:58.375499 15453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:58.375712 15453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:54:58.375808 15453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:54:58.386013 15453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:34989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:34989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:58.387411 15453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:58.388856 15453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:58.401600 15461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:58.401676 15458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:58.401647 15459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:58.402302 15453 server_base.cc:1034] running on GCE node
I20260929 02:54:58.403111 15453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:58.404644 15453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:58.405889 15453 hybrid_clock.cc:648] HybridClock initialized: now 1790650498405846 us; error 33 us; skew 500 ppm
I20260929 02:54:58.406277 15453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:58 dist-test-slave-q5f0 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650498, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:58.409469 15453 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:54:58.410959 15453 webserver.cc:460] Webserver started at http://127.15.18.126:43999/ using document root <none> and password file <none>
I20260929 02:54:58.411702 15453 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:58.411818 15453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:58.412209 15453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:58.415310 15453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "fd22ded0ee5b4f7cbcab969e71681934"
format_stamp: "Formatted at 2026-09-29 02:54:58 on dist-test-slave-q5f0"
server_key: "0e55d7c1fe7c85b533ebb66c0670b513"
server_key_iv: "f68bb6d5e03205372ccc79c6ddad94fd"
server_key_version: "encryptionkey@0"
I20260929 02:54:58.416121 15453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "fd22ded0ee5b4f7cbcab969e71681934"
format_stamp: "Formatted at 2026-09-29 02:54:58 on dist-test-slave-q5f0"
server_key: "0e55d7c1fe7c85b533ebb66c0670b513"
server_key_iv: "f68bb6d5e03205372ccc79c6ddad94fd"
server_key_version: "encryptionkey@0"
I20260929 02:54:58.421531 15453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:54:58.425406 15468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:58.426868 15453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:54:58.427039 15453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "fd22ded0ee5b4f7cbcab969e71681934"
format_stamp: "Formatted at 2026-09-29 02:54:58 on dist-test-slave-q5f0"
server_key: "0e55d7c1fe7c85b533ebb66c0670b513"
server_key_iv: "f68bb6d5e03205372ccc79c6ddad94fd"
server_key_version: "encryptionkey@0"
I20260929 02:54:58.427228 15453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:58.458573 15453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:58.463207 15453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:58.463620 15453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:58.482522 15453 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:34989
I20260929 02:54:58.482530 15519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:34989 every 8 connection(s)
I20260929 02:54:58.483990 15453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:54:58.487918 15520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:54:58.490842 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 15453
I20260929 02:54:58.491161 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:54:58.493695 15433 external_mini_cluster.cc:1428] Setting key 247ffdebd456af9f19c19c462c5a9f39
I20260929 02:54:58.500734 15520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934: Bootstrap starting.
I20260929 02:54:58.503908 15520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934: Neither blocks nor log segments found. Creating new log.
I20260929 02:54:58.504944 15520 log.cc:824] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934: Log is configured to *not* fsync() on all Append() calls
I20260929 02:54:58.507800 15520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934: No bootstrap required, opened a new log
I20260929 02:54:58.517470 15520 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22ded0ee5b4f7cbcab969e71681934" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34989 } }
I20260929 02:54:58.517860 15520 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:54:58.517962 15520 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd22ded0ee5b4f7cbcab969e71681934, State: Initialized, Role: FOLLOWER
I20260929 02:54:58.518694 15520 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22ded0ee5b4f7cbcab969e71681934" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34989 } }
I20260929 02:54:58.518982 15520 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:54:58.519129 15520 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:54:58.519315 15520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:54:58.521651 15520 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22ded0ee5b4f7cbcab969e71681934" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34989 } }
I20260929 02:54:58.522145 15520 leader_election.cc:302] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd22ded0ee5b4f7cbcab969e71681934; no voters:
I20260929 02:54:58.522703 15520 leader_election.cc:288] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:54:58.523033 15525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [term 1 FOLLOWER]: Leader election won for term 1
Sep 29 02:54:58 dist-test-slave-q5f0 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:54:58.524036 15525 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [term 1 LEADER]: Becoming Leader. State: Replica: fd22ded0ee5b4f7cbcab969e71681934, State: Running, Role: LEADER
I20260929 02:54:58.524628 15525 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22ded0ee5b4f7cbcab969e71681934" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34989 } }
I20260929 02:54:58.524971 15520 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:54:58.535589 15527 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd22ded0ee5b4f7cbcab969e71681934. Latest consensus state: current_term: 1 leader_uuid: "fd22ded0ee5b4f7cbcab969e71681934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22ded0ee5b4f7cbcab969e71681934" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34989 } } }
I20260929 02:54:58.536703 15527 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:58.535745 15526 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd22ded0ee5b4f7cbcab969e71681934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22ded0ee5b4f7cbcab969e71681934" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34989 } } }
I20260929 02:54:58.537038 15526 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934 [sys.catalog]: This master's current role is: LEADER
I20260929 02:54:58.538537 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:58.498504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.15.18.126:34989)
0929 02:54:58.499421 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:58.499437 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:58.500771 (+ 1334us) server_negotiation.cc:407] Connection header received
0929 02:54:58.509915 (+ 9144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:58.509944 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:58.510505 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:58.511007 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:58.513902 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:58.516182 (+ 2280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:58.518073 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:58.519095 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:58.519317 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:58.525871 (+ 6554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:58.525882 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:58.525904 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:58.525957 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:58.530451 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:58.531895 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:58.531906 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:58.531915 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:58.532208 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:58.532902 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:58.532907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:58.532910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:58.533651 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:58.533937 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:58.535770 (+ 1833us) server_negotiation.cc:299] Negotiation successful
0929 02:54:58.536622 (+ 852us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":397,"threads_started":1}
I20260929 02:54:58.543984 15538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:54:58.548105 15538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:54:58.556018 15538 catalog_manager.cc:1347] Generated new cluster ID: 7c6edfcf2056446e838ecf39bde1d439
I20260929 02:54:58.556136 15538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:54:58.568675 15538 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:54:58.570623 15538 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:54:58.585541 15538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd22ded0ee5b4f7cbcab969e71681934: Generated new TSK 0
I20260929 02:54:58.586753 15538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:54:58.631700 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34989
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:58.953341 15547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:58.953632 15547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:58.953738 15547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:58.960965 15547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:58.961079 15547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:58.961225 15547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:54:58.970856 15547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34989
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:58.972206 15547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:58.973553 15547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:58.986501 15552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:58.986564 15553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:58.986539 15555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:58.987536 15547 server_base.cc:1034] running on GCE node
I20260929 02:54:58.988168 15547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:58.989069 15547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:58.990276 15547 hybrid_clock.cc:648] HybridClock initialized: now 1790650498990222 us; error 39 us; skew 500 ppm
I20260929 02:54:58.990638 15547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:58 dist-test-slave-q5f0 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650498, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:58.994019 15547 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:54:58.995555 15547 webserver.cc:460] Webserver started at http://127.15.18.65:44121/ using document root <none> and password file <none>
I20260929 02:54:58.996248 15547 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:58.996348 15547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:58.996644 15547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:59.000257 15547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "f53bc71f870f4e1d9b272bc7b75ec6ce"
format_stamp: "Formatted at 2026-09-29 02:54:58 on dist-test-slave-q5f0"
server_key: "589b5d0c255cda5e4092fa5f4211a7d4"
server_key_iv: "e35bc9a1e43c26675509f3ebebdb3e9a"
server_key_version: "encryptionkey@0"
I20260929 02:54:59.001084 15547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "f53bc71f870f4e1d9b272bc7b75ec6ce"
format_stamp: "Formatted at 2026-09-29 02:54:58 on dist-test-slave-q5f0"
server_key: "589b5d0c255cda5e4092fa5f4211a7d4"
server_key_iv: "e35bc9a1e43c26675509f3ebebdb3e9a"
server_key_version: "encryptionkey@0"
I20260929 02:54:59.006495 15547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:54:59.010113 15562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:59.011626 15547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:54:59.011775 15547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "f53bc71f870f4e1d9b272bc7b75ec6ce"
format_stamp: "Formatted at 2026-09-29 02:54:58 on dist-test-slave-q5f0"
server_key: "589b5d0c255cda5e4092fa5f4211a7d4"
server_key_iv: "e35bc9a1e43c26675509f3ebebdb3e9a"
server_key_version: "encryptionkey@0"
I20260929 02:54:59.011957 15547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:59.026592 15547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:59.031010 15547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:59.031284 15547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:59.032366 15547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:54:59.033721 15547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:54:59.033797 15547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:59.033913 15547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:54:59.033968 15547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:59.063606 15547 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:38617
I20260929 02:54:59.063639 15674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:38617 every 8 connection(s)
I20260929 02:54:59.064849 15547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:54:59.069016 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 15547
I20260929 02:54:59.069223 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:54:59.069820 15433 external_mini_cluster.cc:1428] Setting key 72b177260f76f0746ab8d075683b8dfe
Sep 29 02:54:59 dist-test-slave-q5f0 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650498, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
I20260929 02:54:59.085932 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:59.067371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:42529 (local address 127.15.18.126:34989)
0929 02:54:59.067803 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:59.067807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:59.068751 (+ 944us) server_negotiation.cc:407] Connection header received
0929 02:54:59.070615 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:59.070620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:59.070693 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:59.070785 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:59.072422 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:59.073496 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:59.075141 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:59.075696 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:59.075793 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:59.079910 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:59.079915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:59.079918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:59.079954 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:59.082611 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:59.083372 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:59.083377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:59.083379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:59.083505 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:59.083964 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:59.083968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:59.083970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:59.084287 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:59.084452 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:59.085300 (+ 848us) server_negotiation.cc:299] Negotiation successful
0929 02:54:59.085526 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":226,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
I20260929 02:54:59.086824 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:59.067913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34989 (local address 127.15.18.65:42529)
0929 02:54:59.068602 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:59.068640 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:59.070342 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:59.071006 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:59.071014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:59.071664 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:59.072253 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:59.072271 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:59.073681 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:59.073685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:59.075000 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:59.075012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:59.075933 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:59.075940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:59.076249 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:59.076981 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:59.077017 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:59.079681 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:59.082781 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:59.082789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:59.082805 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:59.083237 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:59.083643 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:59.083647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:59.083651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:59.083842 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:59.084407 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:59.084415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:59.084793 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 02:54:59.085040 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 02:54:59.085612 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":246,"threads_started":1}
I20260929 02:54:59.088459 15675 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34989
I20260929 02:54:59.088788 15675 heartbeater.cc:464] Registering TS with master...
I20260929 02:54:59.089495 15675 heartbeater.cc:511] Master 127.15.18.126:34989 requested a full tablet report, sending...
I20260929 02:54:59.091607 15485 ts_manager.cc:194] Registered new tserver with Master: f53bc71f870f4e1d9b272bc7b75ec6ce (127.15.18.65:38617)
I20260929 02:54:59.093019 15485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:42529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:54:59.107663 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34989
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:59.423816 15681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:59.424108 15681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:59.424229 15681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:59.432818 15681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:59.432937 15681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:59.433099 15681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:54:59.443451 15681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34989
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:59.444904 15681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:59.446337 15681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:59.459928 15687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:59.459964 15686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:59.459887 15689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:59.460683 15681 server_base.cc:1034] running on GCE node
I20260929 02:54:59.461277 15681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:59.462282 15681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:59.463500 15681 hybrid_clock.cc:648] HybridClock initialized: now 1790650499463446 us; error 64 us; skew 500 ppm
I20260929 02:54:59.463868 15681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:59 dist-test-slave-q5f0 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:59.467010 15681 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:54:59.468559 15681 webserver.cc:460] Webserver started at http://127.15.18.66:45271/ using document root <none> and password file <none>
I20260929 02:54:59.469233 15681 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:59.469334 15681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:59.469635 15681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:59.472699 15681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "62658c26e1944e2f9d05f83e49510dae"
format_stamp: "Formatted at 2026-09-29 02:54:59 on dist-test-slave-q5f0"
server_key: "d28637c67300c57e3c32f37035627a35"
server_key_iv: "9397a45923616b9228a9fffc6ae15b11"
server_key_version: "encryptionkey@0"
I20260929 02:54:59.473446 15681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "62658c26e1944e2f9d05f83e49510dae"
format_stamp: "Formatted at 2026-09-29 02:54:59 on dist-test-slave-q5f0"
server_key: "d28637c67300c57e3c32f37035627a35"
server_key_iv: "9397a45923616b9228a9fffc6ae15b11"
server_key_version: "encryptionkey@0"
I20260929 02:54:59.478809 15681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:54:59.482702 15696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:59.484513 15681 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:54:59.484694 15681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "62658c26e1944e2f9d05f83e49510dae"
format_stamp: "Formatted at 2026-09-29 02:54:59 on dist-test-slave-q5f0"
server_key: "d28637c67300c57e3c32f37035627a35"
server_key_iv: "9397a45923616b9228a9fffc6ae15b11"
server_key_version: "encryptionkey@0"
I20260929 02:54:59.484917 15681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:54:59.510645 15681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:54:59.514711 15681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:54:59.514954 15681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:54:59.515938 15681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:54:59.517230 15681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:54:59.517304 15681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:59.517392 15681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:54:59.517434 15681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:59.548045 15681 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:44425
I20260929 02:54:59.548146 15808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:44425 every 8 connection(s)
I20260929 02:54:59.549320 15681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:54:59.553754 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 15681
I20260929 02:54:59.553921 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:54:59.554337 15433 external_mini_cluster.cc:1428] Setting key f8ac1dec592aef541618d95a1f48501f
Sep 29 02:54:59 dist-test-slave-q5f0 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
I20260929 02:54:59.570176 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:59.551877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:43931 (local address 127.15.18.126:34989)
0929 02:54:59.552076 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:54:59.552080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:54:59.553314 (+ 1234us) server_negotiation.cc:407] Connection header received
0929 02:54:59.555269 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:54:59.555275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:54:59.555362 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:54:59.555456 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:54:59.557203 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:59.557923 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:59.559493 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:54:59.559949 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:54:59.560063 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:59.564010 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:54:59.564016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:54:59.564019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:54:59.564061 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:54:59.566660 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:59.567568 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:59.567574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:59.567577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:59.567731 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:54:59.568245 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:54:59.568249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:54:59.568251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:54:59.568549 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:54:59.568708 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:54:59.569566 (+ 858us) server_negotiation.cc:299] Negotiation successful
0929 02:54:59.569842 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
I20260929 02:54:59.571436 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:54:59.552377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34989 (local address 127.15.18.66:43931)
0929 02:54:59.553125 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 02:54:59.553179 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:54:59.554982 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:54:59.555751 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:54:59.555760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:54:59.556436 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:54:59.557029 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:59.557047 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:59.558076 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:59.558080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:59.559340 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:54:59.559351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:54:59.560194 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:54:59.560202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:54:59.560428 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:54:59.561137 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:54:59.561161 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:54:59.563718 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:54:59.566860 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:59.566870 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:59.566886 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:59.567397 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:59.567855 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:54:59.567860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:54:59.567864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:54:59.568126 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:54:59.568693 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:54:59.568702 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:54:59.569188 (+ 486us) client_negotiation.cc:769] Sending connection context
0929 02:54:59.569457 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 02:54:59.570103 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":277,"threads_started":1}
I20260929 02:54:59.573036 15809 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34989
I20260929 02:54:59.573342 15809 heartbeater.cc:464] Registering TS with master...
I20260929 02:54:59.574080 15809 heartbeater.cc:511] Master 127.15.18.126:34989 requested a full tablet report, sending...
I20260929 02:54:59.575788 15485 ts_manager.cc:194] Registered new tserver with Master: 62658c26e1944e2f9d05f83e49510dae (127.15.18.66:44425)
I20260929 02:54:59.576572 15485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:43931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:54:59.593088 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34989
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:54:59.909435 15814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:54:59.909780 15814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:54:59.909914 15814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:54:59.917264 15814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:54:59.917393 15814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:54:59.917557 15814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:54:59.927629 15814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34989
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:54:59.928938 15814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:54:59.930349 15814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:54:59.943147 15820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:59.943197 15819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:54:59.943215 15822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:54:59.944026 15814 server_base.cc:1034] running on GCE node
I20260929 02:54:59.944648 15814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:54:59.945607 15814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:54:59.946826 15814 hybrid_clock.cc:648] HybridClock initialized: now 1790650499946778 us; error 47 us; skew 500 ppm
I20260929 02:54:59.947192 15814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:54:59 dist-test-slave-q5f0 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:54:59.950238 15814 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:54:59.951659 15814 webserver.cc:460] Webserver started at http://127.15.18.67:39531/ using document root <none> and password file <none>
I20260929 02:54:59.952337 15814 fs_manager.cc:359] Metadata directory not provided
I20260929 02:54:59.952450 15814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:54:59.952736 15814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:54:59.955632 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "9efd4b3505ec499c89c6fa58c053413a"
format_stamp: "Formatted at 2026-09-29 02:54:59 on dist-test-slave-q5f0"
server_key: "90a06768482eaac6886778fa5149411e"
server_key_iv: "d60bd83ce3127de113b66696a1e3253b"
server_key_version: "encryptionkey@0"
I20260929 02:54:59.956373 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "9efd4b3505ec499c89c6fa58c053413a"
format_stamp: "Formatted at 2026-09-29 02:54:59 on dist-test-slave-q5f0"
server_key: "90a06768482eaac6886778fa5149411e"
server_key_iv: "d60bd83ce3127de113b66696a1e3253b"
server_key_version: "encryptionkey@0"
I20260929 02:54:59.961586 15814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:54:59.965600 15829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:54:59.966988 15814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:54:59.967144 15814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "9efd4b3505ec499c89c6fa58c053413a"
format_stamp: "Formatted at 2026-09-29 02:54:59 on dist-test-slave-q5f0"
server_key: "90a06768482eaac6886778fa5149411e"
server_key_iv: "d60bd83ce3127de113b66696a1e3253b"
server_key_version: "encryptionkey@0"
I20260929 02:54:59.967330 15814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:00.001664 15814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:00.006108 15814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:00.006381 15814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:00.007365 15814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:00.008776 15814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:00.008854 15814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:00.008988 15814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:00.009040 15814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:00.040083 15814 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:37161
I20260929 02:55:00.040153 15941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:37161 every 8 connection(s)
I20260929 02:55:00.041365 15814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:00.049319 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 15814
I20260929 02:55:00.049482 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:00.049917 15433 external_mini_cluster.cc:1428] Setting key ba8a4d42620480eca24d52d07b636b34
Sep 29 02:55:00 dist-test-slave-q5f0 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:00.061416 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:00.043877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:50907 (local address 127.15.18.126:34989)
0929 02:55:00.044072 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:00.044077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:00.045303 (+ 1226us) server_negotiation.cc:407] Connection header received
0929 02:55:00.047090 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:00.047096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:00.047178 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:00.047267 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:00.048870 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.049690 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.051498 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.051948 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.052080 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:00.055778 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:00.055783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:00.055786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:00.055820 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:00.058403 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.059134 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.059141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.059144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.059274 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.059734 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.059739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.059741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.060010 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:00.060151 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:00.060854 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 02:55:00.061080 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:55:00.062424 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:00.044427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34989 (local address 127.15.18.67:50907)
0929 02:55:00.045126 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:00.045170 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:00.046798 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:00.047558 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:00.047571 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:00.048116 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:00.048691 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:00.048709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.049841 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.049845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:00.051354 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:00.051369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.052201 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.052209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:00.052454 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:00.053106 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:00.053133 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:00.055538 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:00.058573 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:00.058581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:00.058595 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:00.059014 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:00.059383 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:00.059387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:00.059389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:00.059628 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:00.060135 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:00.060143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:00.060495 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 02:55:00.060736 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 02:55:00.061272 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":255,"threads_started":1}
I20260929 02:55:00.063889 15942 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34989
I20260929 02:55:00.064218 15942 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:00.065050 15942 heartbeater.cc:511] Master 127.15.18.126:34989 requested a full tablet report, sending...
I20260929 02:55:00.066502 15485 ts_manager.cc:194] Registered new tserver with Master: 9efd4b3505ec499c89c6fa58c053413a (127.15.18.67:37161)
I20260929 02:55:00.067636 15485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:50907
I20260929 02:55:00.078641 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:55:00.096068 15675 heartbeater.cc:503] Master 127.15.18.126:34989 was elected leader, sending a full tablet report...
Sep 29 02:55:00 dist-test-slave-q5f0 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:55:00 dist-test-slave-q5f0 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:00.122326 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:00.106313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.15.18.65:38617)
0929 02:55:00.106889 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:00.106897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:00.106966 (+ 69us) server_negotiation.cc:407] Connection header received
0929 02:55:00.107126 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:00.107133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:00.107353 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:00.107492 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:00.108985 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.110163 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.111450 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.112170 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.112287 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:00.116215 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:00.116230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:00.116245 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:00.116287 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:00.119346 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.120094 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.120105 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.120109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.120294 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.120739 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.120743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.120745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.121149 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:00.121390 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:00.121676 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 02:55:00.121975 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":280,"threads_started":1}
W20260929 02:55:00.123689 15590 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45900
Sep 29 02:55:00 dist-test-slave-q5f0 krb5kdc[15440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:55:00 dist-test-slave-q5f0 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:00.164315 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:00.151599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.15.18.65:38617)
0929 02:55:00.151813 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:00.151819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:00.152006 (+ 187us) server_negotiation.cc:407] Connection header received
0929 02:55:00.152205 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:00.152209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:00.152279 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:00.152369 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:00.153633 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.154313 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.155630 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.156026 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.156122 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:00.159827 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:00.159833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:00.159836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:00.159871 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:00.162151 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.162751 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.162754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.162757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.162878 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.163177 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.163179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.163181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.163413 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:00.163664 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:00.163854 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 02:55:00.164013 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:55:00.166023 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 15547
I20260929 02:55:00.178864 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:00.166962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.15.18.65:38617)
0929 02:55:00.167173 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:00.167178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:00.167379 (+ 201us) server_negotiation.cc:407] Connection header received
0929 02:55:00.167533 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:00.167538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:00.167605 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:00.167683 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:00.168829 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.169525 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.171067 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.171558 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.171668 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:00.174006 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:00.174011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:00.174014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:00.174051 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:00.176521 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.177175 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.177178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.177180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.177303 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.177674 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.177677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.177679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.177924 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:00.178093 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:00.178382 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 02:55:00.178545 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:55:00.179605 15670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:00.263182 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 15547
I20260929 02:55:00.275861 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 15681
Sep 29 02:55:00 dist-test-slave-q5f0 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:55:00.293119 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:00.276750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.15.18.66:44425)
0929 02:55:00.277339 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:00.277346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:00.277374 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:55:00.277436 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:00.277443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:00.277669 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:00.277797 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:00.278986 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.280126 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.281476 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.282062 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.282171 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:00.286325 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:00.286345 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:00.286363 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:00.286415 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:00.289586 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.290551 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.290561 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.290564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.290726 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.291264 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.291268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.291271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.291791 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:00.292046 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:00.292462 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 02:55:00.292712 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":324,"threads_started":1}
I20260929 02:55:00.294405 15804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:00.377550 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 15681
I20260929 02:55:00.390225 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 15814
Sep 29 02:55:00 dist-test-slave-q5f0 krb5kdc[15440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:55:00.406347 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:00.391167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.15.18.67:37161)
0929 02:55:00.391816 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:00.391823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:00.391853 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:55:00.391912 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:00.391919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:00.392132 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:00.392262 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:00.393480 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.394566 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.396021 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:00.396586 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:00.396696 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:00.400753 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:00.400769 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:00.400785 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:00.400823 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:00.403421 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.404183 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.404191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.404193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.404323 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:00.404718 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:00.404722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:00.404724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:00.405083 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:00.405399 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:00.405733 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 02:55:00.405999 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":343,"threads_started":1}
I20260929 02:55:00.407709 15937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:00.490837 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 15814
I20260929 02:55:00.503046 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 15453
I20260929 02:55:00.504262 15515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:00.607801 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 15453
2026-09-29T02:55:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3745 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:00 dist-test-slave-q5f0 krb5kdc[15963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:00 dist-test-slave-q5f0 krb5kdc[15963](info): set up 2 sockets
Sep 29 02:55:00 dist-test-slave-q5f0 krb5kdc[15963](info): commencing operation
krb5kdc: starting...
W20260929 02:55:01.607720 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.009s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:55:01 dist-test-slave-q5f0 krb5kdc[15963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:55:01.779960 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46713
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:46713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:02.092861 15976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:02.093176 15976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:02.093288 15976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:02.100646 15976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:02.100765 15976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:02.100817 15976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:02.100905 15976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:02.100968 15976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:02.111188 15976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:46713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:02.112569 15976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:02.113833 15976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:02.125242 15981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:02.125391 15982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:02.125353 15984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:02.126168 15976 server_base.cc:1034] running on GCE node
I20260929 02:55:02.126987 15976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:02.128511 15976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:02.129753 15976 hybrid_clock.cc:648] HybridClock initialized: now 1790650502129699 us; error 45 us; skew 500 ppm
I20260929 02:55:02.130172 15976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:02 dist-test-slave-q5f0 krb5kdc[15963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650502, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:02.133291 15976 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:55:02.134895 15976 webserver.cc:460] Webserver started at http://127.15.18.126:40171/ using document root <none> and password file <none>
I20260929 02:55:02.135637 15976 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:02.135740 15976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:02.136009 15976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:02.138947 15976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "1ba21d6e156e4865b4a32533fe205d34"
format_stamp: "Formatted at 2026-09-29 02:55:02 on dist-test-slave-q5f0"
server_key: "f2aaf2eba685f77703c0db65cb268b7f"
server_key_iv: "ce3e936e08d1c738dc22937bcaf61ef7"
server_key_version: "encryptionkey@0"
I20260929 02:55:02.139717 15976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "1ba21d6e156e4865b4a32533fe205d34"
format_stamp: "Formatted at 2026-09-29 02:55:02 on dist-test-slave-q5f0"
server_key: "f2aaf2eba685f77703c0db65cb268b7f"
server_key_iv: "ce3e936e08d1c738dc22937bcaf61ef7"
server_key_version: "encryptionkey@0"
I20260929 02:55:02.145008 15976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:55:02.148741 15991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:02.150209 15976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:02.150381 15976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "1ba21d6e156e4865b4a32533fe205d34"
format_stamp: "Formatted at 2026-09-29 02:55:02 on dist-test-slave-q5f0"
server_key: "f2aaf2eba685f77703c0db65cb268b7f"
server_key_iv: "ce3e936e08d1c738dc22937bcaf61ef7"
server_key_version: "encryptionkey@0"
I20260929 02:55:02.150580 15976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:02.187019 15976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:02.191789 15976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:02.192080 15976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:02.209982 15976 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:46713
I20260929 02:55:02.210067 16042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:46713 every 8 connection(s)
I20260929 02:55:02.211410 15976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:02.215191 16043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:02.216835 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 15976
I20260929 02:55:02.217054 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:02.217733 15433 external_mini_cluster.cc:1428] Setting key d880d8c18cafdd5d29eaf14fe10ca155
I20260929 02:55:02.228307 16043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34: Bootstrap starting.
I20260929 02:55:02.231868 16043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:02.232841 16043 log.cc:824] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:55:02 dist-test-slave-q5f0 krb5kdc[15963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:02.236035 16043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34: No bootstrap required, opened a new log
I20260929 02:55:02.242184 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:02.219978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.15.18.126:46713)
0929 02:55:02.220854 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:02.220877 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:02.220943 (+ 66us) server_negotiation.cc:407] Connection header received
0929 02:55:02.223304 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:02.223336 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:02.223942 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:02.224446 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:02.225846 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:02.227790 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:02.229377 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:02.230491 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:02.230656 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:02.234767 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:02.234776 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:02.234792 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:02.234830 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:02.237883 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:02.238737 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:02.238744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:02.238751 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:02.238935 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:02.239313 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:02.239316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:02.239318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:02.239917 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:02.240144 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:02.240673 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 02:55:02.241099 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":352,"threads_started":1}
I20260929 02:55:02.244410 16043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba21d6e156e4865b4a32533fe205d34" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46713 } }
I20260929 02:55:02.244781 16043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:02.244884 16043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba21d6e156e4865b4a32533fe205d34, State: Initialized, Role: FOLLOWER
I20260929 02:55:02.245509 16043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba21d6e156e4865b4a32533fe205d34" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46713 } }
I20260929 02:55:02.245739 16043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:02.245858 16043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:02.246018 16043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:02.248227 16043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba21d6e156e4865b4a32533fe205d34" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46713 } }
I20260929 02:55:02.248747 16043 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba21d6e156e4865b4a32533fe205d34; no voters:
I20260929 02:55:02.249372 16043 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:02.249526 16048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:02.250222 16048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [term 1 LEADER]: Becoming Leader. State: Replica: 1ba21d6e156e4865b4a32533fe205d34, State: Running, Role: LEADER
I20260929 02:55:02.250731 16048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba21d6e156e4865b4a32533fe205d34" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46713 } }
I20260929 02:55:02.251504 16043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:02.253683 16049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba21d6e156e4865b4a32533fe205d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba21d6e156e4865b4a32533fe205d34" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46713 } } }
I20260929 02:55:02.254123 16049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:02.253710 16050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba21d6e156e4865b4a32533fe205d34. Latest consensus state: current_term: 1 leader_uuid: "1ba21d6e156e4865b4a32533fe205d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba21d6e156e4865b4a32533fe205d34" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46713 } } }
I20260929 02:55:02.254304 16050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:02.256013 16057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:02.261492 16057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:02.270418 16057 catalog_manager.cc:1347] Generated new cluster ID: 26786f372fea4488a207eae199a3dc02
I20260929 02:55:02.270545 16057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:02.283188 16057 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:02.284617 16057 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:02.295162 16057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ba21d6e156e4865b4a32533fe205d34: Generated new TSK 0
I20260929 02:55:02.296197 16057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:55:02.335388 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46713
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:02.651221 16069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:02.651543 16069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:02.651686 16069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:02.659269 16069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:02.659377 16069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:02.659492 16069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:02.669353 16069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:02.670634 16069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:02.672001 16069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:02.685091 16074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:02.685117 16075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:02.685243 16077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:02.685859 16069 server_base.cc:1034] running on GCE node
I20260929 02:55:02.686429 16069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:02.687296 16069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:02.688503 16069 hybrid_clock.cc:648] HybridClock initialized: now 1790650502688438 us; error 48 us; skew 500 ppm
I20260929 02:55:02.688858 16069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:02 dist-test-slave-q5f0 krb5kdc[15963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650502, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:02.691983 16069 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:55:02.693451 16069 webserver.cc:460] Webserver started at http://127.15.18.65:38235/ using document root <none> and password file <none>
I20260929 02:55:02.694160 16069 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:02.694263 16069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:02.694547 16069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:02.697695 16069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "1b5c964074dd4c1e9c2553edcec7bc60"
format_stamp: "Formatted at 2026-09-29 02:55:02 on dist-test-slave-q5f0"
server_key: "cf6fc932c9f8f342611d937a19cfb3d4"
server_key_iv: "a86008517f0197132508dec56c6ba4c5"
server_key_version: "encryptionkey@0"
I20260929 02:55:02.698530 16069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "1b5c964074dd4c1e9c2553edcec7bc60"
format_stamp: "Formatted at 2026-09-29 02:55:02 on dist-test-slave-q5f0"
server_key: "cf6fc932c9f8f342611d937a19cfb3d4"
server_key_iv: "a86008517f0197132508dec56c6ba4c5"
server_key_version: "encryptionkey@0"
I20260929 02:55:02.704658 16069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 02:55:02.708376 16084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:02.709851 16069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:02.710045 16069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "1b5c964074dd4c1e9c2553edcec7bc60"
format_stamp: "Formatted at 2026-09-29 02:55:02 on dist-test-slave-q5f0"
server_key: "cf6fc932c9f8f342611d937a19cfb3d4"
server_key_iv: "a86008517f0197132508dec56c6ba4c5"
server_key_version: "encryptionkey@0"
I20260929 02:55:02.710235 16069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:02.729949 16069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:02.734146 16069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:02.734418 16069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:02.735394 16069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:02.736799 16069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:02.736877 16069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:02.737010 16069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:02.737067 16069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:02.766876 16069 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:33387
I20260929 02:55:02.766917 16196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:33387 every 8 connection(s)
I20260929 02:55:02.768189 16069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:02.771299 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 16069
I20260929 02:55:02.771502 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:02.772133 15433 external_mini_cluster.cc:1428] Setting key e545e318e3d2d9684b37b95033e599fe
Sep 29 02:55:02 dist-test-slave-q5f0 krb5kdc[15963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650502, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
I20260929 02:55:02.789499 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:02.770778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:37355 (local address 127.15.18.126:46713)
0929 02:55:02.771281 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:02.771288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:02.772502 (+ 1214us) server_negotiation.cc:407] Connection header received
0929 02:55:02.774355 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:02.774361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:02.774438 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:02.774560 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:02.776200 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:02.777181 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:02.778734 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:02.779219 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:02.779319 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:02.783575 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:02.783581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:02.783584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:02.783618 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:02.786212 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:02.787196 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:02.787202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:02.787204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:02.787333 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:02.787893 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:02.787899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:02.787902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:02.788194 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:02.788312 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:02.789059 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 02:55:02.789237 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":251,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
I20260929 02:55:02.790571 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:02.771371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46713 (local address 127.15.18.65:37355)
0929 02:55:02.772299 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:02.772354 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:02.774084 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:02.774759 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:02.774768 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:02.775359 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:02.776025 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:02.776043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:02.777325 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:02.777328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:02.778583 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:02.778595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:02.779481 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:02.779488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:02.779831 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:02.780589 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:02.780623 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:02.783295 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:02.786386 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:02.786398 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:02.786420 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:02.787044 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:02.787471 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:02.787477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:02.787480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:02.787750 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:02.788317 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:02.788323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:02.788683 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 02:55:02.788977 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 02:55:02.789397 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":326,"threads_started":1}
I20260929 02:55:02.792104 16197 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46713
I20260929 02:55:02.792392 16197 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:02.793083 16197 heartbeater.cc:511] Master 127.15.18.126:46713 requested a full tablet report, sending...
I20260929 02:55:02.795241 16008 ts_manager.cc:194] Registered new tserver with Master: 1b5c964074dd4c1e9c2553edcec7bc60 (127.15.18.65:33387)
I20260929 02:55:02.797303 16008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:37355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:55:02.811107 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46713
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:03.125697 16203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:03.126001 16203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:03.126147 16203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:03.133688 16203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:03.133828 16203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:03.133988 16203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:03.143924 16203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:03.145390 16203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:03.146770 16203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:03.161038 16209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:03.161093 16208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:03.161091 16211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:03.161679 16203 server_base.cc:1034] running on GCE node
I20260929 02:55:03.162338 16203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:03.163394 16203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:03.164631 16203 hybrid_clock.cc:648] HybridClock initialized: now 1790650503164577 us; error 46 us; skew 500 ppm
I20260929 02:55:03.164994 16203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:03 dist-test-slave-q5f0 krb5kdc[15963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:03.168309 16203 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:55:03.169796 16203 webserver.cc:460] Webserver started at http://127.15.18.66:37163/ using document root <none> and password file <none>
I20260929 02:55:03.170493 16203 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:03.170596 16203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:03.170893 16203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:03.173830 16203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "cc57b9982cc14da3a2bac5e0b3814a6a"
format_stamp: "Formatted at 2026-09-29 02:55:03 on dist-test-slave-q5f0"
server_key: "41c65f8f8a749792b998ce6231db5f48"
server_key_iv: "5ff033a6683d670a0d4ed2c3f46bc86d"
server_key_version: "encryptionkey@0"
I20260929 02:55:03.174580 16203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "cc57b9982cc14da3a2bac5e0b3814a6a"
format_stamp: "Formatted at 2026-09-29 02:55:03 on dist-test-slave-q5f0"
server_key: "41c65f8f8a749792b998ce6231db5f48"
server_key_iv: "5ff033a6683d670a0d4ed2c3f46bc86d"
server_key_version: "encryptionkey@0"
I20260929 02:55:03.179721 16203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:55:03.183115 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:03.184440 16203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:03.184584 16203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "cc57b9982cc14da3a2bac5e0b3814a6a"
format_stamp: "Formatted at 2026-09-29 02:55:03 on dist-test-slave-q5f0"
server_key: "41c65f8f8a749792b998ce6231db5f48"
server_key_iv: "5ff033a6683d670a0d4ed2c3f46bc86d"
server_key_version: "encryptionkey@0"
I20260929 02:55:03.184770 16203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:03.204216 16203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:03.208912 16203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:03.209192 16203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:03.210286 16203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:03.211784 16203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:03.211879 16203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:03.212007 16203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:03.212062 16203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:03.243714 16203 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:43219
I20260929 02:55:03.243791 16330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:43219 every 8 connection(s)
I20260929 02:55:03.245005 16203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:03.247102 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 16203
I20260929 02:55:03.247298 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:03.247891 15433 external_mini_cluster.cc:1428] Setting key 6bec75a5a05ebdb893b2e4481bf17562
Sep 29 02:55:03 dist-test-slave-q5f0 krb5kdc[15963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
I20260929 02:55:03.266850 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.247859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:38403 (local address 127.15.18.126:46713)
0929 02:55:03.248045 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:03.248051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:03.249506 (+ 1455us) server_negotiation.cc:407] Connection header received
0929 02:55:03.251450 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:03.251456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:03.251568 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:03.251658 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:03.253499 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.254397 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.256200 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.256606 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.256716 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.260399 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:03.260404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:03.260407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:03.260442 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:03.263254 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:03.264131 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:03.264136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:03.264138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:03.264300 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:03.264743 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:03.264747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:03.264749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:03.265238 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:03.265437 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:03.266236 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 02:55:03.266514 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:55:03.267683 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.248579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46713 (local address 127.15.18.66:38403)
0929 02:55:03.249346 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:03.249392 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:03.251120 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:03.251930 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:03.251951 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:03.252650 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:03.253301 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.253322 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.254548 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.254553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.256061 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.256074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.256837 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.256845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.257078 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.257783 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:03.257811 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:03.260160 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:03.263415 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:03.263425 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:03.263444 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:03.263992 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:03.264439 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:03.264442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:03.264445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:03.264638 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:03.265401 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:03.265410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:03.265804 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 02:55:03.266057 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 02:55:03.266497 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":306,"threads_started":1}
I20260929 02:55:03.269075 16331 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46713
I20260929 02:55:03.269418 16331 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:03.270200 16331 heartbeater.cc:511] Master 127.15.18.126:46713 requested a full tablet report, sending...
I20260929 02:55:03.271636 16008 ts_manager.cc:194] Registered new tserver with Master: cc57b9982cc14da3a2bac5e0b3814a6a (127.15.18.66:43219)
I20260929 02:55:03.272398 16008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:38403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:55:03.286161 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46713
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:03.607628 16336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:03.607987 16336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:03.608126 16336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:03.615588 16336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:03.615724 16336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:03.615882 16336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:03.625777 16336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:03.627099 16336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:03.628455 16336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:03.641713 16341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:03.641743 16342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:03.641685 16344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:03.642590 16336 server_base.cc:1034] running on GCE node
I20260929 02:55:03.643206 16336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:03.644291 16336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:03.645521 16336 hybrid_clock.cc:648] HybridClock initialized: now 1790650503645464 us; error 47 us; skew 500 ppm
I20260929 02:55:03.645830 16336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:03 dist-test-slave-q5f0 krb5kdc[15963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:03.654644 16336 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:55:03.656404 16336 webserver.cc:460] Webserver started at http://127.15.18.67:39915/ using document root <none> and password file <none>
I20260929 02:55:03.657101 16336 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:03.657207 16336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:03.657469 16336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:03.660449 16336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "54bed899e15f4bcb892d733659b6cbc1"
format_stamp: "Formatted at 2026-09-29 02:55:03 on dist-test-slave-q5f0"
server_key: "3bf7d42928ceb2db270566574078576a"
server_key_iv: "42177105dd92252dd4b26613df4ef47c"
server_key_version: "encryptionkey@0"
I20260929 02:55:03.661232 16336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "54bed899e15f4bcb892d733659b6cbc1"
format_stamp: "Formatted at 2026-09-29 02:55:03 on dist-test-slave-q5f0"
server_key: "3bf7d42928ceb2db270566574078576a"
server_key_iv: "42177105dd92252dd4b26613df4ef47c"
server_key_version: "encryptionkey@0"
I20260929 02:55:03.666544 16336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:55:03.670195 16351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:03.671844 16336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:03.672012 16336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "54bed899e15f4bcb892d733659b6cbc1"
format_stamp: "Formatted at 2026-09-29 02:55:03 on dist-test-slave-q5f0"
server_key: "3bf7d42928ceb2db270566574078576a"
server_key_iv: "42177105dd92252dd4b26613df4ef47c"
server_key_version: "encryptionkey@0"
I20260929 02:55:03.672230 16336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:03.695178 16336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:03.699465 16336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:03.699790 16336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:03.700805 16336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:03.702163 16336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:03.702240 16336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:03.702337 16336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:03.702391 16336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:03.733048 16336 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:46871
I20260929 02:55:03.733194 16463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:46871 every 8 connection(s)
I20260929 02:55:03.734364 16336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:03.742841 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 16336
I20260929 02:55:03.743064 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:03.743635 15433 external_mini_cluster.cc:1428] Setting key 11ddfe0302e498f10d2f4c7d6a527d40
Sep 29 02:55:03 dist-test-slave-q5f0 krb5kdc[15963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:03.756459 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.736986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:42013 (local address 127.15.18.126:46713)
0929 02:55:03.737165 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:03.737169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:03.738392 (+ 1223us) server_negotiation.cc:407] Connection header received
0929 02:55:03.740586 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:03.740592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:03.740679 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:03.740755 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:03.742608 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.743711 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.745609 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.746255 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.746379 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.750397 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:03.750403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:03.750407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:03.750446 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:03.753068 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:03.753891 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:03.753896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:03.753899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:03.754038 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:03.754566 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:03.754571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:03.754575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:03.754897 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:03.755081 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:03.755946 (+ 865us) server_negotiation.cc:299] Negotiation successful
0929 02:55:03.756206 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:55:03.757445 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.737495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46713 (local address 127.15.18.67:42013)
0929 02:55:03.738228 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:03.738266 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:03.740243 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:03.741282 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:03.741294 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:03.741839 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:03.742426 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.742450 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.743868 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.743873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.745474 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.745487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.746522 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.746530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.746829 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.747699 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:03.747730 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:03.750136 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:03.753235 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:03.753243 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:03.753258 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:03.753721 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:03.754198 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:03.754203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:03.754207 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:03.754420 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:03.755091 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:03.755099 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:03.755492 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 02:55:03.755826 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 02:55:03.756280 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":281,"threads_started":1}
I20260929 02:55:03.758944 16464 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46713
I20260929 02:55:03.759269 16464 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:03.760057 16464 heartbeater.cc:511] Master 127.15.18.126:46713 requested a full tablet report, sending...
I20260929 02:55:03.761510 16007 ts_manager.cc:194] Registered new tserver with Master: 54bed899e15f4bcb892d733659b6cbc1 (127.15.18.67:46871)
I20260929 02:55:03.762367 16007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:42013
I20260929 02:55:03.772418 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:55:03.791342 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.778770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.15.18.126:46713)
0929 02:55:03.778952 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:03.778957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:03.779394 (+ 437us) server_negotiation.cc:407] Connection header received
0929 02:55:03.779607 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:03.779612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:03.779677 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:03.779766 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:03.781219 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.781969 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.783271 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.783723 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.783817 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.786084 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:03.786089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:03.786092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:03.786126 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:03.788673 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:03.789425 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:03.789429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:03.789432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:03.789607 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:03.789953 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:03.789956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:03.789958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:03.790312 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:03.790508 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:03.790899 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 02:55:03.791098 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:55:03.800491 16197 heartbeater.cc:503] Master 127.15.18.126:46713 was elected leader, sending a full tablet report...
I20260929 02:55:03.798301 16007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:03.805528 16007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:03.846115 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.818588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:33387 (local address 127.0.0.1:58494)
0929 02:55:03.834835 (+ 16247us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:03.834887 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:03.835109 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:03.838090 (+ 2981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:03.838104 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:03.838133 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:03.838565 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.838580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.840760 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.840768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.843217 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.843235 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.845130 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.845137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.845338 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.845388 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 02:55:03.845532 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 02:55:03.845734 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15889,"thread_start_us":15692,"threads_started":1}
I20260929 02:55:03.846093 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.818959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.15.18.65:33387)
0929 02:55:03.836884 (+ 17925us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:03.836894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:03.836931 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:55:03.837143 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:03.837156 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:03.837471 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:03.837714 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:03.838782 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.840556 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.843413 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.844927 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.845071 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.845314 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:03.845579 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 02:55:03.845753 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17717,"thread_start_us":16210,"threads_started":1}
I20260929 02:55:03.850035 16132 tablet_service.cc:1467] Processing CreateTablet for tablet abc9cc9672c949a1b131eac6d412ba62 (DEFAULT_TABLE table=test-table [id=f5ae9a6d39c545b999b13d3416c05cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:03.851855 16132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc9cc9672c949a1b131eac6d412ba62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:03.852455 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.821812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:43219 (local address 127.0.0.1:54564)
0929 02:55:03.843739 (+ 21927us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:03.843756 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:03.843875 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:03.844578 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:03.844583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:03.844607 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:03.844984 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.844996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.847601 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.847606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.849836 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.849853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.851621 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.851628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.851790 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.851823 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:55:03.851934 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:55:03.852073 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21814,"mutex_wait_us":58,"thread_start_us":21039,"threads_started":1}
I20260929 02:55:03.852744 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.819050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46871 (local address 127.0.0.1:50440)
0929 02:55:03.842569 (+ 23519us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:03.842594 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:03.842753 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:03.844244 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:03.844250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:03.844279 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:03.844695 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.844712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.846568 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.846574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.849835 (+ 3261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.849853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.851947 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.851954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.852126 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.852159 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:55:03.852243 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 02:55:03.852375 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23335,"mutex_wait_us":107,"thread_start_us":24863,"threads_started":1}
I20260929 02:55:03.853171 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.821945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.15.18.66:43219)
0929 02:55:03.842803 (+ 20858us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:03.842813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:03.843873 (+ 1060us) server_negotiation.cc:407] Connection header received
0929 02:55:03.844076 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:03.844084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:03.844318 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:03.844503 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:03.845168 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.847438 (+ 2270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.850056 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.851125 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.851252 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.851451 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:03.852752 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0929 02:55:03.852949 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19055,"thread_start_us":20596,"threads_started":1}
I20260929 02:55:03.853608 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.819218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.15.18.67:46871)
0929 02:55:03.843568 (+ 24350us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:03.843577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:03.843613 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:55:03.843716 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:03.843725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:03.844034 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:03.844248 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:03.844935 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.846425 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.850048 (+ 3623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.851421 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.851588 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.853053 (+ 1465us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:03.853125 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 02:55:03.853287 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24126,"thread_start_us":23979,"threads_started":1}
I20260929 02:55:03.857188 16266 tablet_service.cc:1467] Processing CreateTablet for tablet abc9cc9672c949a1b131eac6d412ba62 (DEFAULT_TABLE table=test-table [id=f5ae9a6d39c545b999b13d3416c05cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:03.857190 16399 tablet_service.cc:1467] Processing CreateTablet for tablet abc9cc9672c949a1b131eac6d412ba62 (DEFAULT_TABLE table=test-table [id=f5ae9a6d39c545b999b13d3416c05cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:03.858666 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc9cc9672c949a1b131eac6d412ba62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:03.859287 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc9cc9672c949a1b131eac6d412ba62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:03.864081 16482 tablet_bootstrap.cc:492] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60: Bootstrap starting.
I20260929 02:55:03.867715 16482 tablet_bootstrap.cc:654] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:03.868721 16482 log.cc:824] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:03.873831 16482 tablet_bootstrap.cc:492] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60: No bootstrap required, opened a new log
I20260929 02:55:03.874121 16482 ts_tablet_manager.cc:1397] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260929 02:55:03.875479 16484 tablet_bootstrap.cc:492] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a: Bootstrap starting.
I20260929 02:55:03.879272 16484 tablet_bootstrap.cc:654] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:03.880105 16485 tablet_bootstrap.cc:492] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1: Bootstrap starting.
I20260929 02:55:03.880645 16484 log.cc:824] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:03.883471 16485 tablet_bootstrap.cc:654] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:03.884059 16482 raft_consensus.cc:348] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 33387 } } peers { permanent_uuid: "54bed899e15f4bcb892d733659b6cbc1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46871 } } peers { permanent_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43219 } }
I20260929 02:55:03.884428 16484 tablet_bootstrap.cc:492] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a: No bootstrap required, opened a new log
I20260929 02:55:03.884420 16482 raft_consensus.cc:374] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:03.884512 16482 raft_consensus.cc:729] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5c964074dd4c1e9c2553edcec7bc60, State: Initialized, Role: FOLLOWER
I20260929 02:55:03.884708 16484 ts_tablet_manager.cc:1397] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 02:55:03.884774 16485 log.cc:824] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:03.885088 16482 consensus_queue.cc:260] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 33387 } } peers { permanent_uuid: "54bed899e15f4bcb892d733659b6cbc1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46871 } } peers { permanent_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43219 } }
I20260929 02:55:03.888460 16482 ts_tablet_manager.cc:1428] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260929 02:55:03.888707 16485 tablet_bootstrap.cc:492] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1: No bootstrap required, opened a new log
I20260929 02:55:03.889045 16485 ts_tablet_manager.cc:1397] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:55:03.895063 16484 raft_consensus.cc:348] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 33387 } } peers { permanent_uuid: "54bed899e15f4bcb892d733659b6cbc1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46871 } } peers { permanent_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43219 } }
I20260929 02:55:03.895630 16484 raft_consensus.cc:374] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:03.895779 16484 raft_consensus.cc:729] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc57b9982cc14da3a2bac5e0b3814a6a, State: Initialized, Role: FOLLOWER
I20260929 02:55:03.896385 16484 consensus_queue.cc:260] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 33387 } } peers { permanent_uuid: "54bed899e15f4bcb892d733659b6cbc1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46871 } } peers { permanent_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43219 } }
I20260929 02:55:03.897521 16331 heartbeater.cc:503] Master 127.15.18.126:46713 was elected leader, sending a full tablet report...
I20260929 02:55:03.897507 16485 raft_consensus.cc:348] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 33387 } } peers { permanent_uuid: "54bed899e15f4bcb892d733659b6cbc1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46871 } } peers { permanent_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43219 } }
I20260929 02:55:03.897985 16485 raft_consensus.cc:374] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:03.898087 16485 raft_consensus.cc:729] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bed899e15f4bcb892d733659b6cbc1, State: Initialized, Role: FOLLOWER
I20260929 02:55:03.898161 16484 ts_tablet_manager.cc:1428] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260929 02:55:03.898730 16485 consensus_queue.cc:260] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 33387 } } peers { permanent_uuid: "54bed899e15f4bcb892d733659b6cbc1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46871 } } peers { permanent_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43219 } }
I20260929 02:55:03.899933 16464 heartbeater.cc:503] Master 127.15.18.126:46713 was elected leader, sending a full tablet report...
I20260929 02:55:03.900691 16485 ts_tablet_manager.cc:1428] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 02:55:03.906095 16488 raft_consensus.cc:482] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:03.906353 16488 raft_consensus.cc:504] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 33387 } } peers { permanent_uuid: "54bed899e15f4bcb892d733659b6cbc1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46871 } } peers { permanent_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43219 } }
I20260929 02:55:03.908057 16488 leader_election.cc:288] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54bed899e15f4bcb892d733659b6cbc1 (127.15.18.67:46871), cc57b9982cc14da3a2bac5e0b3814a6a (127.15.18.66:43219)
I20260929 02:55:03.914268 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.908305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:42687 (local address 127.15.18.67:46871)
0929 02:55:03.908503 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:03.908510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:03.909169 (+ 659us) server_negotiation.cc:407] Connection header received
0929 02:55:03.909350 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:03.909354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:03.909411 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:03.909479 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:03.910009 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.910724 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.912500 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.913270 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.913423 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.913499 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:03.913878 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 02:55:03.914031 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:55:03.914579 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.908504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:46129 (local address 127.15.18.66:43219)
0929 02:55:03.908727 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:03.908733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:03.909362 (+ 629us) server_negotiation.cc:407] Connection header received
0929 02:55:03.909527 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:03.909530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:03.909585 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:03.909669 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:03.910225 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.910919 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.912658 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.913656 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.913802 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.913937 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:03.914227 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 02:55:03.914355 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:55:03.914631 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.908266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46871 (local address 127.15.18.65:42687)
0929 02:55:03.909060 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:03.909080 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:03.909224 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:03.909505 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:03.909509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:03.909533 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:03.909879 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.909888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.910842 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.910847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.912355 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.912367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.913461 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.913467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.913616 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.913640 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:55:03.913854 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 02:55:03.914007 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":4,"thread_start_us":680,"threads_started":1}
I20260929 02:55:03.915025 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.908331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:43219 (local address 127.15.18.65:46129)
0929 02:55:03.909267 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:03.909280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:03.909398 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:03.909707 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:03.909716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:03.909740 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:03.910089 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.910098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.911054 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.911061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.912505 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:03.912517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.913908 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.913915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:03.914081 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.914104 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:55:03.914187 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 02:55:03.914310 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":18,"thread_start_us":418,"threads_started":1}
I20260929 02:55:03.915642 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc9cc9672c949a1b131eac6d412ba62" candidate_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bed899e15f4bcb892d733659b6cbc1" is_pre_election: true
I20260929 02:55:03.915925 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc9cc9672c949a1b131eac6d412ba62" candidate_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" is_pre_election: true
I20260929 02:55:03.916201 16419 raft_consensus.cc:2454] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b5c964074dd4c1e9c2553edcec7bc60 in term 0.
I20260929 02:55:03.916389 16286 raft_consensus.cc:2454] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b5c964074dd4c1e9c2553edcec7bc60 in term 0.
I20260929 02:55:03.916941 16086 leader_election.cc:302] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5c964074dd4c1e9c2553edcec7bc60, 54bed899e15f4bcb892d733659b6cbc1; no voters:
I20260929 02:55:03.917318 16488 raft_consensus.cc:2781] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:03.917451 16488 raft_consensus.cc:482] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:03.917547 16488 raft_consensus.cc:3037] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:03.919673 16488 raft_consensus.cc:504] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 33387 } } peers { permanent_uuid: "54bed899e15f4bcb892d733659b6cbc1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46871 } } peers { permanent_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43219 } }
I20260929 02:55:03.920215 16488 leader_election.cc:288] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [CANDIDATE]: Term 1 election: Requested vote from peers 54bed899e15f4bcb892d733659b6cbc1 (127.15.18.67:46871), cc57b9982cc14da3a2bac5e0b3814a6a (127.15.18.66:43219)
I20260929 02:55:03.920650 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc9cc9672c949a1b131eac6d412ba62" candidate_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bed899e15f4bcb892d733659b6cbc1"
I20260929 02:55:03.920660 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc9cc9672c949a1b131eac6d412ba62" candidate_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a"
I20260929 02:55:03.920822 16419 raft_consensus.cc:3037] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:03.920827 16286 raft_consensus.cc:3037] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:03.922969 16419 raft_consensus.cc:2454] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b5c964074dd4c1e9c2553edcec7bc60 in term 1.
I20260929 02:55:03.922953 16286 raft_consensus.cc:2454] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b5c964074dd4c1e9c2553edcec7bc60 in term 1.
I20260929 02:55:03.923368 16086 leader_election.cc:302] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5c964074dd4c1e9c2553edcec7bc60, 54bed899e15f4bcb892d733659b6cbc1; no voters:
I20260929 02:55:03.923653 16488 raft_consensus.cc:2781] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:03.924031 16488 raft_consensus.cc:686] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [term 1 LEADER]: Becoming Leader. State: Replica: 1b5c964074dd4c1e9c2553edcec7bc60, State: Running, Role: LEADER
I20260929 02:55:03.924474 16488 consensus_queue.cc:237] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 33387 } } peers { permanent_uuid: "54bed899e15f4bcb892d733659b6cbc1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46871 } } peers { permanent_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43219 } }
I20260929 02:55:03.928930 16007 catalog_manager.cc:5495] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b5c964074dd4c1e9c2553edcec7bc60 (127.15.18.65). New cstate: current_term: 1 leader_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5c964074dd4c1e9c2553edcec7bc60" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 33387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54bed899e15f4bcb892d733659b6cbc1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc57b9982cc14da3a2bac5e0b3814a6a" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43219 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:55:03 dist-test-slave-q5f0 krb5kdc[15963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:03.952813 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.938388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.15.18.65:33387)
0929 02:55:03.938633 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:03.938640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:03.939019 (+ 379us) server_negotiation.cc:407] Connection header received
0929 02:55:03.939158 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:03.939162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:03.939239 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:03.939317 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:03.940744 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.941465 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.942721 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.943111 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.943202 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.947075 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:03.947092 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:03.947131 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:03.947183 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:03.950244 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:03.950899 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:03.950907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:03.950909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:03.951078 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:03.951392 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:03.951395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:03.951397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:03.951778 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:03.952022 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:03.952279 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 02:55:03.952479 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 02:55:03 dist-test-slave-q5f0 krb5kdc[15963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 02:55:03.985946 16465 tablet.cc:2369] T abc9cc9672c949a1b131eac6d412ba62 P 54bed899e15f4bcb892d733659b6cbc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:55:03 dist-test-slave-q5f0 krb5kdc[15963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:03.994947 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:03.981966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.15.18.65:33387)
0929 02:55:03.982172 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:03.982179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:03.982507 (+ 328us) server_negotiation.cc:407] Connection header received
0929 02:55:03.982688 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:03.982694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:03.982763 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:03.982846 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:03.984155 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.984885 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.986094 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:03.986528 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:03.986635 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:03.990406 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:03.990412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:03.990416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:03.990450 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:03.992811 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:03.993374 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:03.993378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:03.993380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:03.993502 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:03.993801 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:03.993804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:03.993806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:03.994054 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:03.994269 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:03.994453 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 02:55:03.994624 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 02:55:03.997164 16332 tablet.cc:2369] T abc9cc9672c949a1b131eac6d412ba62 P cc57b9982cc14da3a2bac5e0b3814a6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:55:04 dist-test-slave-q5f0 krb5kdc[15963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260929 02:55:04.020581 16198 tablet.cc:2369] T abc9cc9672c949a1b131eac6d412ba62 P 1b5c964074dd4c1e9c2553edcec7bc60: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:55:04 dist-test-slave-q5f0 krb5kdc[15963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:04.037441 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:04.023391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.15.18.65:33387)
0929 02:55:04.023645 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:04.023651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:04.023888 (+ 237us) server_negotiation.cc:407] Connection header received
0929 02:55:04.024030 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:04.024034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:04.024113 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:04.024199 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:04.025566 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:04.026363 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:04.027633 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:04.028058 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:04.028157 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:04.032078 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:04.032083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:04.032087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:04.032128 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:04.034818 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:04.035499 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:04.035504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:04.035525 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:04.035673 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:04.036051 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:04.036055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:04.036057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:04.036336 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:04.036578 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:04.036874 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 02:55:04.037102 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260929 02:55:04.039269 16112 tablet_service.cc:2908] Rejecting scan request for tablet abc9cc9672c949a1b131eac6d412ba62: Uninitialized: clean time has not yet been initialized
I20260929 02:55:04.042156 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 16069
I20260929 02:55:04.054504 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:04.043130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.15.18.65:33387)
0929 02:55:04.043290 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:04.043294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:04.043621 (+ 327us) server_negotiation.cc:407] Connection header received
0929 02:55:04.043767 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:04.043771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:04.043831 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:04.043910 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:04.045102 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:04.045811 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:04.047005 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:04.047407 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:04.047495 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:04.049564 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:04.049569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:04.049572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:04.049614 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:04.052118 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:04.052813 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:04.052817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:04.052819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:04.052939 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:04.053278 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:04.053281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:04.053283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:04.053507 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:04.053683 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:04.054011 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 02:55:04.054250 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:55:04.055136 16192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:04.144847 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 16069
I20260929 02:55:04.158663 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 16203
Sep 29 02:55:04 dist-test-slave-q5f0 krb5kdc[15963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:55:04.173899 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:04.159429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.15.18.66:43219)
0929 02:55:04.159622 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:04.159626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:04.159644 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:04.159832 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:04.159837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:04.159905 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:04.159996 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:04.161227 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:04.162036 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:04.163354 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:04.163935 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:04.164045 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:04.168016 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:04.168033 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:04.168049 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:04.168090 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:04.170871 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:04.171744 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:04.171754 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:04.171757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:04.171916 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:04.172312 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:04.172315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:04.172317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:04.172735 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:04.173046 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:04.173374 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 02:55:04.173600 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:55:04.174479 16326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:04.261169 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 16203
I20260929 02:55:04.274258 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 16336
Sep 29 02:55:04 dist-test-slave-q5f0 krb5kdc[15963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:55:04.289660 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:04.274993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.15.18.67:46871)
0929 02:55:04.275179 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:04.275183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:04.275201 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:04.275352 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:04.275355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:04.275426 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:04.275523 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:04.276779 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:04.277648 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:04.278887 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:04.279294 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:04.279380 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:04.283604 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:04.283623 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:04.283640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:04.283682 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:04.286525 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:04.287359 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:04.287367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:04.287370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:04.287577 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:04.288050 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:04.288054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:04.288057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:04.288499 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:04.288728 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:04.289126 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 02:55:04.289307 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:55:04.290345 16459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:04.381916 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 16336
I20260929 02:55:04.394573 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 15976
I20260929 02:55:04.395908 16038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:04.502746 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 15976
2026-09-29T02:55:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3894 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:04 dist-test-slave-q5f0 krb5kdc[16512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:04 dist-test-slave-q5f0 krb5kdc[16512](info): set up 2 sockets
Sep 29 02:55:04 dist-test-slave-q5f0 krb5kdc[16512](info): commencing operation
krb5kdc: starting...
W20260929 02:55:05.507807 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:55:05 dist-test-slave-q5f0 krb5kdc[16512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:55:05.682839 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:32979
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:32979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:05.993995 16525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:05.994333 16525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:05.994444 16525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:06.001813 16525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:06.001935 16525 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:55:06.001977 16525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:06.002070 16525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:06.002133 16525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:06.002218 16525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:06.012224 16525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:32979
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:06.013505 16525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:06.014854 16525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:06.026255 16530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:06.026326 16531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:06.026288 16533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:06.027814 16525 server_base.cc:1034] running on GCE node
I20260929 02:55:06.028623 16525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:06.030117 16525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:06.031347 16525 hybrid_clock.cc:648] HybridClock initialized: now 1790650506031276 us; error 51 us; skew 500 ppm
I20260929 02:55:06.031710 16525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:06 dist-test-slave-q5f0 krb5kdc[16512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:06.034770 16525 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:55:06.036170 16525 webserver.cc:460] Webserver started at http://127.15.18.126:46681/ using document root <none> and password file <none>
I20260929 02:55:06.036856 16525 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:06.036952 16525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:06.037284 16525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:06.040292 16525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "8f0b13048f364eee80dd40f225f5b0f3"
format_stamp: "Formatted at 2026-09-29 02:55:06 on dist-test-slave-q5f0"
server_key: "d49f1e17c97968f4a893153bf5288903"
server_key_iv: "4a520591d00198ac4071ded61aec8fdd"
server_key_version: "encryptionkey@0"
I20260929 02:55:06.041024 16525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "8f0b13048f364eee80dd40f225f5b0f3"
format_stamp: "Formatted at 2026-09-29 02:55:06 on dist-test-slave-q5f0"
server_key: "d49f1e17c97968f4a893153bf5288903"
server_key_iv: "4a520591d00198ac4071ded61aec8fdd"
server_key_version: "encryptionkey@0"
I20260929 02:55:06.046058 16525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:55:06.049530 16540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:06.050897 16525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:55:06.051059 16525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "8f0b13048f364eee80dd40f225f5b0f3"
format_stamp: "Formatted at 2026-09-29 02:55:06 on dist-test-slave-q5f0"
server_key: "d49f1e17c97968f4a893153bf5288903"
server_key_iv: "4a520591d00198ac4071ded61aec8fdd"
server_key_version: "encryptionkey@0"
I20260929 02:55:06.051256 16525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:06.074366 16525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:06.078701 16525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:06.078946 16525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:06.099278 16525 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:32979
I20260929 02:55:06.099299 16601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:32979 every 8 connection(s)
I20260929 02:55:06.100703 16525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:06.104610 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:06.109339 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 16525
I20260929 02:55:06.109612 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:06.110430 15433 external_mini_cluster.cc:1428] Setting key feb5343de35342de82b93f11df02a329
I20260929 02:55:06.116578 16602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3: Bootstrap starting.
I20260929 02:55:06.120287 16602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:06.121424 16602 log.cc:824] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:06.124732 16602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3: No bootstrap required, opened a new log
Sep 29 02:55:06 dist-test-slave-q5f0 krb5kdc[16512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:06.134656 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:06.112602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.15.18.126:32979)
0929 02:55:06.113443 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:06.113456 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:06.113501 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:55:06.115285 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:06.115314 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:06.115796 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:06.116143 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:06.117837 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:06.119188 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:06.120861 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:06.121546 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:06.121704 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:06.126096 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:06.126106 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:06.126126 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:06.126173 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:06.130021 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:06.130724 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:06.130734 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:06.130744 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:06.130961 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:06.131307 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:06.131312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:06.131315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:06.132099 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:06.132389 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:06.132795 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 02:55:06.133254 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":395,"threads_started":1}
I20260929 02:55:06.135037 16602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0b13048f364eee80dd40f225f5b0f3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 32979 } }
I20260929 02:55:06.135376 16602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:06.135473 16602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0b13048f364eee80dd40f225f5b0f3, State: Initialized, Role: FOLLOWER
I20260929 02:55:06.136073 16602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0b13048f364eee80dd40f225f5b0f3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 32979 } }
I20260929 02:55:06.136307 16602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:06.136452 16602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:06.136615 16602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:06.138860 16602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0b13048f364eee80dd40f225f5b0f3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 32979 } }
I20260929 02:55:06.139350 16602 leader_election.cc:302] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0b13048f364eee80dd40f225f5b0f3; no voters:
I20260929 02:55:06.139909 16602 leader_election.cc:288] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:06.140051 16607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:06.140710 16607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [term 1 LEADER]: Becoming Leader. State: Replica: 8f0b13048f364eee80dd40f225f5b0f3, State: Running, Role: LEADER
I20260929 02:55:06.141234 16607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0b13048f364eee80dd40f225f5b0f3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 32979 } }
I20260929 02:55:06.141811 16602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:06.144706 16609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0b13048f364eee80dd40f225f5b0f3. Latest consensus state: current_term: 1 leader_uuid: "8f0b13048f364eee80dd40f225f5b0f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0b13048f364eee80dd40f225f5b0f3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 32979 } } }
I20260929 02:55:06.144785 16608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f0b13048f364eee80dd40f225f5b0f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0b13048f364eee80dd40f225f5b0f3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 32979 } } }
I20260929 02:55:06.145077 16609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:06.145110 16608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:06.147177 16615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:06.152661 16615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:06.161989 16615 catalog_manager.cc:1347] Generated new cluster ID: 5538ca0a926f46daab76f084fa706895
I20260929 02:55:06.162128 16615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:06.173089 16615 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:06.174530 16615 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:06.183139 16615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3: Generated new TSK 0
I20260929 02:55:06.184350 16615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:55:06.236397 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:32979
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:06.552264 16628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:06.552564 16628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:06.552693 16628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:06.560205 16628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:06.560319 16628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:06.560432 16628 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:55:06.560520 16628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:06.570494 16628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.18.126:32979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:06.571887 16628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:06.573184 16628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:06.585829 16633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:06.585747 16636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:06.585796 16634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:06.592123 16628 server_base.cc:1034] running on GCE node
I20260929 02:55:06.592764 16628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:06.593726 16628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:06.594983 16628 hybrid_clock.cc:648] HybridClock initialized: now 1790650506594913 us; error 50 us; skew 500 ppm
I20260929 02:55:06.595289 16628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:06 dist-test-slave-q5f0 krb5kdc[16512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:06.598748 16628 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:55:06.600307 16628 webserver.cc:460] Webserver started at http://127.15.18.65:43345/ using document root <none> and password file <none>
I20260929 02:55:06.601011 16628 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:06.601097 16628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:06.601372 16628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:06.604317 16628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "922cf3d5bd73482c8b13e51b22eabd58"
format_stamp: "Formatted at 2026-09-29 02:55:06 on dist-test-slave-q5f0"
server_key: "90755f99c90610fcd51bed199b0607e2"
server_key_iv: "2727fd20bc01d90e4ea51a1cba4e908b"
server_key_version: "encryptionkey@0"
I20260929 02:55:06.605223 16628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "922cf3d5bd73482c8b13e51b22eabd58"
format_stamp: "Formatted at 2026-09-29 02:55:06 on dist-test-slave-q5f0"
server_key: "90755f99c90610fcd51bed199b0607e2"
server_key_iv: "2727fd20bc01d90e4ea51a1cba4e908b"
server_key_version: "encryptionkey@0"
I20260929 02:55:06.610311 16628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:06.613718 16643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:06.615069 16628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:55:06.615217 16628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "922cf3d5bd73482c8b13e51b22eabd58"
format_stamp: "Formatted at 2026-09-29 02:55:06 on dist-test-slave-q5f0"
server_key: "90755f99c90610fcd51bed199b0607e2"
server_key_iv: "2727fd20bc01d90e4ea51a1cba4e908b"
server_key_version: "encryptionkey@0"
I20260929 02:55:06.615404 16628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:06.643639 16628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:06.648100 16628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:06.648347 16628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:06.650928 16628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:06.651067 16628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:06.651214 16628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:06.651289 16628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:55:06 dist-test-slave-q5f0 krb5kdc[16512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:06.674839 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:06.651719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:41287 (local address 127.15.18.126:32979)
0929 02:55:06.654189 (+ 2470us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:06.654195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:06.654219 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:55:06.656794 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:06.656800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:06.656895 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:06.657000 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:06.659355 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:06.660353 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:06.662759 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:06.663330 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:06.663460 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:06.668447 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:06.668453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:06.668456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:06.668493 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:06.670872 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:06.671664 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:06.671670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:06.671674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:06.671835 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:06.672437 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:06.672441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:06.672444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:06.672797 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:06.673008 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:06.674185 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0929 02:55:06.674451 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2331,"thread_start_us":2222,"threads_started":1}
I20260929 02:55:06.676599 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:06.652404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:32979 (local address 127.15.18.65:41287)
0929 02:55:06.653844 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:06.653906 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:06.656453 (+ 2547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:06.657297 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:06.657316 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:06.658204 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:06.659148 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:06.659178 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:06.660575 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:06.660581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:06.662584 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:06.662605 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:06.663602 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:06.663610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:06.663972 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:06.664794 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:06.664817 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:06.668165 (+ 3348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:06.671032 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:06.671043 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:06.671062 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:06.671546 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:06.671978 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:06.671986 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:06.671990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:06.672306 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:06.673013 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:06.673024 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:06.673631 (+ 607us) client_negotiation.cc:769] Sending connection context
0929 02:55:06.674062 (+ 431us) client_negotiation.cc:240] Negotiation successful
0929 02:55:06.674709 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":406,"threads_started":1}
I20260929 02:55:06.698295 16628 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:34351
I20260929 02:55:06.698309 16763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:34351 every 8 connection(s)
I20260929 02:55:06.699921 16628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:06.699954 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:06.686085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:32979 (local address 127.15.18.65:42637)
0929 02:55:06.686671 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:06.686696 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:06.686820 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:06.687193 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:06.687196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:06.687680 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:06.688156 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:06.688178 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:06.689511 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:06.689516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:06.690741 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:06.690753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:06.691761 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:06.691767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:06.691978 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:06.692620 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:06.692644 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:06.693788 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:06.697180 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:06.697185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:06.697190 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:06.697770 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:06.698177 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:06.698186 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:06.698189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:06.698356 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:06.698999 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:06.699003 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:06.699178 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 02:55:06.699309 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 02:55:06.699625 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":302,"threads_started":1}
I20260929 02:55:06.700011 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:06.686077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:42637 (local address 127.15.18.126:32979)
0929 02:55:06.686278 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:06.686285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:06.686773 (+ 488us) server_negotiation.cc:407] Connection header received
0929 02:55:06.686936 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:06.686942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:06.687032 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:06.687149 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:06.688380 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:06.689354 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:06.690879 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:06.691351 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:06.691486 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:06.693973 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:06.693978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:06.693981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:06.694014 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:06.697037 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:06.697886 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:06.697892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:06.697896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:06.698048 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:06.698508 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:06.698514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:06.698517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:06.698859 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:06.699018 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:06.699429 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 02:55:06.699689 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:55:06.701876 16764 heartbeater.cc:347] Connected to a master server at 127.15.18.126:32979
I20260929 02:55:06.702098 16764 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:06.702801 16764 heartbeater.cc:511] Master 127.15.18.126:32979 requested a full tablet report, sending...
I20260929 02:55:06.703631 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 16628
I20260929 02:55:06.703850 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:06.704445 15433 external_mini_cluster.cc:1428] Setting key ba5f75b3e32c3ad6ff31c733b12c2dc8
I20260929 02:55:06.704819 16557 ts_manager.cc:194] Registered new tserver with Master: 922cf3d5bd73482c8b13e51b22eabd58 (127.15.18.65:34351)
I20260929 02:55:06.706593 16557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:41287
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:55:06.743433 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:32979
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:07.053288 16768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:07.053583 16768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:07.053722 16768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:07.060994 16768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:07.061131 16768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:07.061262 16768 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:55:07.061375 16768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:07.071110 16768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.18.126:32979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:07.072443 16768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:07.073814 16768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:07.086444 16773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:07.086969 16774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:07.086987 16776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:07.087329 16768 server_base.cc:1034] running on GCE node
I20260929 02:55:07.087978 16768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:07.088891 16768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:07.090122 16768 hybrid_clock.cc:648] HybridClock initialized: now 1790650507090034 us; error 62 us; skew 500 ppm
I20260929 02:55:07.090446 16768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:07 dist-test-slave-q5f0 krb5kdc[16512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650507, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:07.094089 16768 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:55:07.095674 16768 webserver.cc:460] Webserver started at http://127.15.18.66:34123/ using document root <none> and password file <none>
I20260929 02:55:07.096383 16768 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:07.096481 16768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:07.096789 16768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:07.099858 16768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "f503956960dc410fa0c8aaf472570b80"
format_stamp: "Formatted at 2026-09-29 02:55:07 on dist-test-slave-q5f0"
server_key: "937bb472b2bf7a472966e8af154b290a"
server_key_iv: "348f6fb1613064ed3288e27eeb5616c5"
server_key_version: "encryptionkey@0"
I20260929 02:55:07.100659 16768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "f503956960dc410fa0c8aaf472570b80"
format_stamp: "Formatted at 2026-09-29 02:55:07 on dist-test-slave-q5f0"
server_key: "937bb472b2bf7a472966e8af154b290a"
server_key_iv: "348f6fb1613064ed3288e27eeb5616c5"
server_key_version: "encryptionkey@0"
I20260929 02:55:07.105865 16768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:55:07.109342 16783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:07.110707 16768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:55:07.110850 16768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "f503956960dc410fa0c8aaf472570b80"
format_stamp: "Formatted at 2026-09-29 02:55:07 on dist-test-slave-q5f0"
server_key: "937bb472b2bf7a472966e8af154b290a"
server_key_iv: "348f6fb1613064ed3288e27eeb5616c5"
server_key_version: "encryptionkey@0"
I20260929 02:55:07.111050 16768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:07.124763 16768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:07.128931 16768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:07.129223 16768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:07.131938 16768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:07.132056 16768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:07.132184 16768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:07.132257 16768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:55:07 dist-test-slave-q5f0 krb5kdc[16512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650507, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:07.155822 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.132599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:43361 (local address 127.15.18.126:32979)
0929 02:55:07.132799 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.132805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.134437 (+ 1632us) server_negotiation.cc:407] Connection header received
0929 02:55:07.136577 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.136583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.136670 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.136767 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:07.139031 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.140205 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.143119 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.143699 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.143840 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.148811 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:07.148818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:07.148823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:07.148877 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:07.151821 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:07.152798 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:07.152804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:07.152808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:07.152958 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:07.153635 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:07.153641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:07.153645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:07.154053 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:07.154214 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.155125 (+ 911us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.155380 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:55:07.156781 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.133355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:32979 (local address 127.15.18.66:43361)
0929 02:55:07.134234 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.134280 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.136233 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.137074 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.137090 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.137897 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:07.138823 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.138851 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.140384 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.140393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.142926 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.142945 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.143966 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.143973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.144201 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.144984 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:07.145019 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:07.148518 (+ 3499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:07.152008 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:07.152025 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:07.152048 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:07.152679 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:07.153104 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:07.153112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:07.153117 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:07.153451 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:07.154244 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:07.154250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:07.154708 (+ 458us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.155004 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.155420 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":381,"threads_started":1}
I20260929 02:55:07.177309 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.162213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:49017 (local address 127.15.18.126:32979)
0929 02:55:07.162373 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.162379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.163033 (+ 654us) server_negotiation.cc:407] Connection header received
0929 02:55:07.163172 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.163177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.163264 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.163362 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:07.164552 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.165319 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.167547 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.167968 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.168078 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.170496 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:07.170502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:07.170505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:07.170540 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:07.173921 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:07.174757 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:07.174764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:07.174768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:07.174960 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:07.175469 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:07.175475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:07.175479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:07.175954 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:07.176129 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.176657 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.176913 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:55:07.177500 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.162251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:32979 (local address 127.15.18.66:49017)
0929 02:55:07.162928 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.162943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.163062 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.163393 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.163395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.163972 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:07.164399 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.164409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.165475 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.165487 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.167350 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.167368 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.168156 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.168163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.168342 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.169120 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:07.169145 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:07.170276 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:07.174090 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:07.174094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:07.174097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:07.174605 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:07.175118 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:07.175124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:07.175131 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:07.175344 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:07.176093 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:07.176098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:07.176333 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.176526 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.176929 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":397,"threads_started":1}
I20260929 02:55:07.184003 16768 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:46009
I20260929 02:55:07.184072 16902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:46009 every 8 connection(s)
I20260929 02:55:07.185247 16768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:07.187629 16903 heartbeater.cc:347] Connected to a master server at 127.15.18.126:32979
I20260929 02:55:07.187885 16903 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:07.188614 16903 heartbeater.cc:511] Master 127.15.18.126:32979 requested a full tablet report, sending...
I20260929 02:55:07.190056 16557 ts_manager.cc:194] Registered new tserver with Master: f503956960dc410fa0c8aaf472570b80 (127.15.18.66:46009)
I20260929 02:55:07.190342 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 16768
I20260929 02:55:07.190521 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:07.190937 15433 external_mini_cluster.cc:1428] Setting key b9519e589895506d034cc2853f610320
I20260929 02:55:07.190933 16557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:43361
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:55:07.231474 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:32979
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:07.558637 16907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:07.558952 16907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:07.559119 16907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:07.566476 16907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:07.566619 16907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:07.566754 16907 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:55:07.566848 16907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:07.576741 16907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.18.126:32979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:07.578042 16907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:07.579427 16907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:07.592957 16915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:07.592983 16912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:07.593281 16913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:07.593745 16907 server_base.cc:1034] running on GCE node
I20260929 02:55:07.594359 16907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:07.595306 16907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:07.596524 16907 hybrid_clock.cc:648] HybridClock initialized: now 1790650507596443 us; error 56 us; skew 500 ppm
I20260929 02:55:07.596844 16907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:07 dist-test-slave-q5f0 krb5kdc[16512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650507, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:07.600137 16907 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:55:07.601576 16907 webserver.cc:460] Webserver started at http://127.15.18.67:44933/ using document root <none> and password file <none>
I20260929 02:55:07.602242 16907 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:07.602330 16907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:07.602603 16907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:07.605588 16907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "fe2684924680452c9bc5ceb47a7c1787"
format_stamp: "Formatted at 2026-09-29 02:55:07 on dist-test-slave-q5f0"
server_key: "1fb21d3e4b9dcb94b965f14c49fcb053"
server_key_iv: "4d0a8d76585c2acc9e2442dfbba6e558"
server_key_version: "encryptionkey@0"
I20260929 02:55:07.606349 16907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "fe2684924680452c9bc5ceb47a7c1787"
format_stamp: "Formatted at 2026-09-29 02:55:07 on dist-test-slave-q5f0"
server_key: "1fb21d3e4b9dcb94b965f14c49fcb053"
server_key_iv: "4d0a8d76585c2acc9e2442dfbba6e558"
server_key_version: "encryptionkey@0"
I20260929 02:55:07.611446 16907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:07.615027 16922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:07.616492 16907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:07.616657 16907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "fe2684924680452c9bc5ceb47a7c1787"
format_stamp: "Formatted at 2026-09-29 02:55:07 on dist-test-slave-q5f0"
server_key: "1fb21d3e4b9dcb94b965f14c49fcb053"
server_key_iv: "4d0a8d76585c2acc9e2442dfbba6e558"
server_key_version: "encryptionkey@0"
I20260929 02:55:07.616864 16907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:07.628247 16907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:07.632314 16907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:07.632586 16907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:07.635237 16907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:07.635329 16907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:07.635465 16907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:07.635543 16907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:55:07 dist-test-slave-q5f0 krb5kdc[16512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650507, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:07.660265 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.636048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:59169 (local address 127.15.18.126:32979)
0929 02:55:07.636274 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.636280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.637711 (+ 1431us) server_negotiation.cc:407] Connection header received
0929 02:55:07.640406 (+ 2695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.640412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.640525 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.640622 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:07.643233 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.644425 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.646728 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.647338 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.647477 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.652796 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:07.652804 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:07.652809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:07.652865 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:07.656249 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:07.657162 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:07.657167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:07.657170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:07.657332 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:07.657937 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:07.657942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:07.657946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:07.658331 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:07.658486 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.659700 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.659930 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:55:07.661613 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.636642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:32979 (local address 127.15.18.67:59169)
0929 02:55:07.637500 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.637548 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.640009 (+ 2461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.640971 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.640986 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.641917 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:07.642919 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.642944 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.644624 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.644633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.646568 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.646585 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.647728 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.647737 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.648113 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.649021 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:07.649076 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:07.652556 (+ 3480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:07.656404 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:07.656416 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:07.656437 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:07.657030 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:07.657509 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:07.657515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:07.657519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:07.657816 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:07.658472 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:07.658480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:07.659010 (+ 530us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.659366 (+ 356us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.659989 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":361,"threads_started":1}
I20260929 02:55:07.673149 16907 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:41977
I20260929 02:55:07.673169 17041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:41977 every 8 connection(s)
I20260929 02:55:07.674603 16907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:07.677670 17042 heartbeater.cc:347] Connected to a master server at 127.15.18.126:32979
I20260929 02:55:07.677903 17042 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:07.678553 17042 heartbeater.cc:511] Master 127.15.18.126:32979 requested a full tablet report, sending...
I20260929 02:55:07.678810 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 16907
I20260929 02:55:07.678992 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:07.679548 15433 external_mini_cluster.cc:1428] Setting key 3598371461b7e1be934fdb6663d69a79
I20260929 02:55:07.680065 16557 ts_manager.cc:194] Registered new tserver with Master: fe2684924680452c9bc5ceb47a7c1787 (127.15.18.67:41977)
I20260929 02:55:07.681221 16557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:59169
I20260929 02:55:07.682703 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:55:07.683157 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.668603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:46339 (local address 127.15.18.126:32979)
0929 02:55:07.668767 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.668771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.669383 (+ 612us) server_negotiation.cc:407] Connection header received
0929 02:55:07.669564 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.669570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.669663 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.669764 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:07.670997 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.671801 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.673881 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.674280 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.674381 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.676481 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:07.676486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:07.676490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:07.676521 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:07.679795 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:07.680595 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:07.680600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:07.680604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:07.680775 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:07.681147 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:07.681151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:07.681154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:07.681491 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:07.681651 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.682586 (+ 935us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.682822 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:55:07.688182 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.668716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:32979 (local address 127.15.18.67:46339)
0929 02:55:07.669276 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.669291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.669436 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.669822 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.669831 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.670444 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:07.670822 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.670836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.672021 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.672030 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.673708 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.673723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.674465 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.674472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.674646 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.675280 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:07.675304 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:07.676291 (+ 987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:07.679962 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:07.679967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:07.679971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:07.680460 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:07.680878 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:07.680883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:07.680887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:07.681034 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:07.682058 (+ 1024us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:07.682064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:07.682285 (+ 221us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.682444 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.687850 (+ 5406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":5042,"thread_start_us":291,"threads_started":1}
Sep 29 02:55:07 dist-test-slave-q5f0 krb5kdc[16512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650507, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260929 02:55:07.709319 16764 heartbeater.cc:503] Master 127.15.18.126:32979 was elected leader, sending a full tablet report...
Sep 29 02:55:07 dist-test-slave-q5f0 krb5kdc[16512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650507, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:07.726608 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.713270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45822 (local address 127.15.18.126:32979)
0929 02:55:07.713489 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.713496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.713745 (+ 249us) server_negotiation.cc:407] Connection header received
0929 02:55:07.713917 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.713921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.713985 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.714058 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:07.715402 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.716210 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.717470 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.717872 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.717960 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.721902 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:07.721907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:07.721911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:07.721944 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:07.724189 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:07.724790 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:07.724794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:07.724797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:07.724910 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:07.725258 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:07.725263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:07.725267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:07.725530 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:07.725754 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.726042 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.726259 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:55:07.730754 16556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:07.737497 16556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:07.769724 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.750210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.15.18.67:41977)
0929 02:55:07.762171 (+ 11961us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.762181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.762219 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:55:07.762331 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.762339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.762633 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.762819 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:07.763630 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.765287 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.767659 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.768602 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.768719 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.768960 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.769236 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.769391 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11776,"thread_start_us":12835,"threads_started":1}
I20260929 02:55:07.769851 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.749596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:41977 (local address 127.0.0.1:46528)
0929 02:55:07.750262 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.750309 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.750500 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.762852 (+ 12352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.762860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.762907 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:07.763331 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.763345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.765422 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.765427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.767481 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.767493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.768810 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.768816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.769019 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.769060 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.769170 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.769366 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":18,"thread_start_us":817,"threads_started":1}
I20260929 02:55:07.772495 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.749707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32786 (local address 127.15.18.66:46009)
0929 02:55:07.764110 (+ 14403us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.764119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.764147 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:55:07.764291 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.764299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.764546 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.764763 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:07.765732 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.767206 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.770089 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.771402 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.771560 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.771760 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.772046 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.772213 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14253,"thread_start_us":14138,"threads_started":1}
I20260929 02:55:07.772853 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.749622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:46009 (local address 127.0.0.1:32786)
0929 02:55:07.761566 (+ 11944us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.761588 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.761725 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.765053 (+ 3328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.765058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.765084 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:07.765585 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.765596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.767377 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.767382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.769929 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.769940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.771681 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.771687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.771856 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.771892 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.771989 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.772129 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11771,"mutex_wait_us":101,"thread_start_us":14602,"threads_started":1}
I20260929 02:55:07.773612 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.753495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.15.18.65:34351)
0929 02:55:07.763725 (+ 10230us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.763735 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.764555 (+ 820us) server_negotiation.cc:407] Connection header received
0929 02:55:07.764777 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.764782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.764945 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.765107 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:07.765908 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.767307 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.770039 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.772357 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.772488 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.772674 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.773196 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.773352 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10059,"thread_start_us":9891,"threads_started":1}
I20260929 02:55:07.773661 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.752963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:34351 (local address 127.0.0.1:48254)
0929 02:55:07.764437 (+ 11474us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.764452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.764575 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.765116 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.765120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.765148 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:07.765722 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.765739 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.767491 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.767496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.769857 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.769871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.772868 (+ 2997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.772876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.773037 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.773060 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.773138 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.773264 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11366,"mutex_wait_us":81,"thread_start_us":8271,"threads_started":1}
I20260929 02:55:07.774070 16972 tablet_service.cc:1467] Processing CreateTablet for tablet 21fb01c83b464f9baf4fee5ca2906973 (DEFAULT_TABLE table=test-table [id=1353677845014aa3900167651fbaa7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:07.775573 16972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fb01c83b464f9baf4fee5ca2906973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:07.777581 16833 tablet_service.cc:1467] Processing CreateTablet for tablet 21fb01c83b464f9baf4fee5ca2906973 (DEFAULT_TABLE table=test-table [id=1353677845014aa3900167651fbaa7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:07.777551 16694 tablet_service.cc:1467] Processing CreateTablet for tablet 21fb01c83b464f9baf4fee5ca2906973 (DEFAULT_TABLE table=test-table [id=1353677845014aa3900167651fbaa7ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:07.778903 16833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fb01c83b464f9baf4fee5ca2906973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:07.778910 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fb01c83b464f9baf4fee5ca2906973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:07.791640 17062 tablet_bootstrap.cc:492] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: Bootstrap starting.
I20260929 02:55:07.793901 17061 tablet_bootstrap.cc:492] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: Bootstrap starting.
I20260929 02:55:07.795624 17063 tablet_bootstrap.cc:492] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: Bootstrap starting.
I20260929 02:55:07.795804 17062 tablet_bootstrap.cc:654] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:07.797266 17062 log.cc:824] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:07.797840 17061 tablet_bootstrap.cc:654] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:07.798717 17063 tablet_bootstrap.cc:654] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:07.799114 17061 log.cc:824] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:07.799790 17063 log.cc:824] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:07.801155 17062 tablet_bootstrap.cc:492] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: No bootstrap required, opened a new log
I20260929 02:55:07.801484 17062 ts_tablet_manager.cc:1397] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260929 02:55:07.802390 17061 tablet_bootstrap.cc:492] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: No bootstrap required, opened a new log
I20260929 02:55:07.802676 17061 ts_tablet_manager.cc:1397] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:55:07.808537 17063 tablet_bootstrap.cc:492] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: No bootstrap required, opened a new log
I20260929 02:55:07.808808 17063 ts_tablet_manager.cc:1397] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260929 02:55:07.812772 17062 raft_consensus.cc:348] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.813313 17062 raft_consensus.cc:374] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:07.813760 17062 raft_consensus.cc:729] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f503956960dc410fa0c8aaf472570b80, State: Initialized, Role: FOLLOWER
I20260929 02:55:07.813478 17061 raft_consensus.cc:348] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.813947 17061 raft_consensus.cc:374] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:07.814070 17061 raft_consensus.cc:729] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2684924680452c9bc5ceb47a7c1787, State: Initialized, Role: FOLLOWER
I20260929 02:55:07.814626 17062 consensus_queue.cc:260] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.814705 17061 consensus_queue.cc:260] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.816102 16903 heartbeater.cc:503] Master 127.15.18.126:32979 was elected leader, sending a full tablet report...
I20260929 02:55:07.816994 17062 ts_tablet_manager.cc:1428] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 02:55:07.816843 17063 raft_consensus.cc:348] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.817278 17063 raft_consensus.cc:374] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:07.817363 17063 raft_consensus.cc:729] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922cf3d5bd73482c8b13e51b22eabd58, State: Initialized, Role: FOLLOWER
I20260929 02:55:07.817947 17063 consensus_queue.cc:260] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.818041 17042 heartbeater.cc:503] Master 127.15.18.126:32979 was elected leader, sending a full tablet report...
I20260929 02:55:07.818711 17061 ts_tablet_manager.cc:1428] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260929 02:55:07.819795 17063 ts_tablet_manager.cc:1428] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 02:55:07.835855 17068 raft_consensus.cc:482] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:07.836180 17068 raft_consensus.cc:504] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.836450 17067 raft_consensus.cc:482] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:07.836759 17067 raft_consensus.cc:504] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.838277 17068 leader_election.cc:288] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe2684924680452c9bc5ceb47a7c1787 (127.15.18.67:41977), 922cf3d5bd73482c8b13e51b22eabd58 (127.15.18.65:34351)
I20260929 02:55:07.844877 17067 leader_election.cc:288] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f503956960dc410fa0c8aaf472570b80 (127.15.18.66:46009), 922cf3d5bd73482c8b13e51b22eabd58 (127.15.18.65:34351)
I20260929 02:55:07.853432 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.838893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:34351 (local address 127.15.18.67:55017)
0929 02:55:07.839201 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.839218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.839340 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.845984 (+ 6644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.845990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.846046 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:07.846286 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.846298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.849097 (+ 2799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.849103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.851042 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.851054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.852628 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.852637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.852836 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.852865 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.852926 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.853120 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":45}
I20260929 02:55:07.853489 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.838643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:41977 (local address 127.15.18.66:55621)
0929 02:55:07.844831 (+ 6188us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.844857 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.845542 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.846164 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.846169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.846202 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:07.846580 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.846593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.848397 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.848403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.850768 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.850784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.852587 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.852593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.852776 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.852811 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.853014 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.853154 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6061,"thread_start_us":5868,"threads_started":1}
I20260929 02:55:07.853670 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.839662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:55017 (local address 127.15.18.65:34351)
0929 02:55:07.845638 (+ 5976us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.845644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.845666 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:55:07.845739 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.845745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.845819 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.846437 (+ 618us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:07.846742 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.848932 (+ 2190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.851233 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.852461 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.853071 (+ 610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.853138 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.853184 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.853309 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5856,"thread_start_us":5674,"threads_started":1}
I20260929 02:55:07.854272 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.838925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:55621 (local address 127.15.18.67:41977)
0929 02:55:07.839117 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.839123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.844957 (+ 5834us) server_negotiation.cc:407] Connection header received
0929 02:55:07.845728 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.845735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.845822 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.845922 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:07.846758 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.848146 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.850963 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.852176 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.852304 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.852379 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.852959 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.854001 (+ 1042us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":875,"server-negotiator.queue_time_us":69,"spinlock_wait_cycles":49920}
I20260929 02:55:07.854789 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fb01c83b464f9baf4fee5ca2906973" candidate_uuid: "fe2684924680452c9bc5ceb47a7c1787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922cf3d5bd73482c8b13e51b22eabd58" is_pre_election: true
I20260929 02:55:07.855391 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.838714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:57451 (local address 127.15.18.65:34351)
0929 02:55:07.838952 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.838959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.848735 (+ 9776us) server_negotiation.cc:407] Connection header received
0929 02:55:07.848975 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.848980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.849057 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.849115 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:07.849957 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.850994 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.853147 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.854342 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.854484 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.854547 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.855043 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.855168 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:55:07.855212 16992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fb01c83b464f9baf4fee5ca2906973" candidate_uuid: "f503956960dc410fa0c8aaf472570b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe2684924680452c9bc5ceb47a7c1787" is_pre_election: true
I20260929 02:55:07.855396 16714 raft_consensus.cc:2454] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe2684924680452c9bc5ceb47a7c1787 in term 0.
I20260929 02:55:07.855698 16992 raft_consensus.cc:2454] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f503956960dc410fa0c8aaf472570b80 in term 0.
I20260929 02:55:07.856082 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.839441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:34351 (local address 127.15.18.66:57451)
0929 02:55:07.848651 (+ 9210us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.848667 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.848813 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.849288 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.849292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.849319 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:07.849750 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.849765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.851148 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.851152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.852990 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.853003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.854684 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.854690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.854854 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.854889 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.855618 (+ 729us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.855760 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9076,"thread_start_us":8838,"threads_started":1}
I20260929 02:55:07.856284 16924 leader_election.cc:302] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 922cf3d5bd73482c8b13e51b22eabd58, fe2684924680452c9bc5ceb47a7c1787; no voters:
I20260929 02:55:07.856738 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fb01c83b464f9baf4fee5ca2906973" candidate_uuid: "f503956960dc410fa0c8aaf472570b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922cf3d5bd73482c8b13e51b22eabd58" is_pre_election: true
I20260929 02:55:07.856822 16784 leader_election.cc:302] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f503956960dc410fa0c8aaf472570b80, fe2684924680452c9bc5ceb47a7c1787; no voters:
I20260929 02:55:07.857074 16714 raft_consensus.cc:2454] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f503956960dc410fa0c8aaf472570b80 in term 0.
I20260929 02:55:07.857239 17068 raft_consensus.cc:2781] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:07.857301 17067 raft_consensus.cc:2781] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:07.857393 17068 raft_consensus.cc:482] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:07.857417 17067 raft_consensus.cc:482] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:07.857504 17068 raft_consensus.cc:3037] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:07.857519 17067 raft_consensus.cc:3037] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:07.858870 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.840001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:50291 (local address 127.15.18.66:46009)
0929 02:55:07.840197 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.840203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.848419 (+ 8216us) server_negotiation.cc:407] Connection header received
0929 02:55:07.848690 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.848695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.848764 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.848825 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:07.849714 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.850896 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.856576 (+ 5680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.857929 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.858077 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.858151 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.858490 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.858629 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:55:07.859114 17071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.839439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:46009 (local address 127.15.18.67:50291)
0929 02:55:07.848304 (+ 8865us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:07.848326 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:07.848534 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:07.849004 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:07.849008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:07.849032 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:07.849499 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.849511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.852587 (+ 3076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.852592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.856397 (+ 3805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:07.856426 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.858178 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.858186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:07.858309 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.858335 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 02:55:07.858420 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 02:55:07.858578 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8720,"thread_start_us":8526,"threads_started":1}
I20260929 02:55:07.860129 16858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fb01c83b464f9baf4fee5ca2906973" candidate_uuid: "fe2684924680452c9bc5ceb47a7c1787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f503956960dc410fa0c8aaf472570b80" is_pre_election: true
I20260929 02:55:07.860783 17067 raft_consensus.cc:504] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.860766 17068 raft_consensus.cc:504] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.861238 16858 raft_consensus.cc:2379] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe2684924680452c9bc5ceb47a7c1787 in current term 1: Already voted for candidate f503956960dc410fa0c8aaf472570b80 in this term.
I20260929 02:55:07.861804 17068 leader_election.cc:288] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [CANDIDATE]: Term 1 election: Requested vote from peers fe2684924680452c9bc5ceb47a7c1787 (127.15.18.67:41977), 922cf3d5bd73482c8b13e51b22eabd58 (127.15.18.65:34351)
I20260929 02:55:07.861815 16992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fb01c83b464f9baf4fee5ca2906973" candidate_uuid: "f503956960dc410fa0c8aaf472570b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe2684924680452c9bc5ceb47a7c1787"
I20260929 02:55:07.862087 16992 raft_consensus.cc:2379] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f503956960dc410fa0c8aaf472570b80 in current term 1: Already voted for candidate fe2684924680452c9bc5ceb47a7c1787 in this term.
I20260929 02:55:07.862106 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fb01c83b464f9baf4fee5ca2906973" candidate_uuid: "fe2684924680452c9bc5ceb47a7c1787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922cf3d5bd73482c8b13e51b22eabd58"
I20260929 02:55:07.862161 16713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fb01c83b464f9baf4fee5ca2906973" candidate_uuid: "f503956960dc410fa0c8aaf472570b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922cf3d5bd73482c8b13e51b22eabd58"
I20260929 02:55:07.862329 16714 raft_consensus.cc:3037] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:07.862936 17067 leader_election.cc:288] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [CANDIDATE]: Term 1 election: Requested vote from peers f503956960dc410fa0c8aaf472570b80 (127.15.18.66:46009), 922cf3d5bd73482c8b13e51b22eabd58 (127.15.18.65:34351)
I20260929 02:55:07.863014 16858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fb01c83b464f9baf4fee5ca2906973" candidate_uuid: "fe2684924680452c9bc5ceb47a7c1787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f503956960dc410fa0c8aaf472570b80"
I20260929 02:55:07.863224 16858 raft_consensus.cc:2379] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fe2684924680452c9bc5ceb47a7c1787 in current term 1: Already voted for candidate f503956960dc410fa0c8aaf472570b80 in this term.
I20260929 02:55:07.863569 16785 leader_election.cc:302] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f503956960dc410fa0c8aaf472570b80; no voters: 922cf3d5bd73482c8b13e51b22eabd58, fe2684924680452c9bc5ceb47a7c1787
I20260929 02:55:07.863862 17068 raft_consensus.cc:2726] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 02:55:07.865131 16714 raft_consensus.cc:2454] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe2684924680452c9bc5ceb47a7c1787 in term 1.
I20260929 02:55:07.865545 16924 leader_election.cc:302] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 922cf3d5bd73482c8b13e51b22eabd58, fe2684924680452c9bc5ceb47a7c1787; no voters: f503956960dc410fa0c8aaf472570b80
I20260929 02:55:07.865834 17067 raft_consensus.cc:2781] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:07.866262 17067 raft_consensus.cc:686] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 1 LEADER]: Becoming Leader. State: Replica: fe2684924680452c9bc5ceb47a7c1787, State: Running, Role: LEADER
I20260929 02:55:07.866811 17067 consensus_queue.cc:237] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } }
I20260929 02:55:07.871217 16556 catalog_manager.cc:5495] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe2684924680452c9bc5ceb47a7c1787 (127.15.18.67). New cstate: current_term: 1 leader_uuid: "fe2684924680452c9bc5ceb47a7c1787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } health_report { overall_health: UNKNOWN } } }
I20260929 02:55:07.895623 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:07.890111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.15.18.67:41977)
0929 02:55:07.890318 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:07.890325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:07.890435 (+ 110us) server_negotiation.cc:407] Connection header received
0929 02:55:07.890743 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:07.890747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:07.890844 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:07.891002 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:07.891367 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.892163 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.893473 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:07.893854 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:07.893970 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:07.894431 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:07.894968 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:07.895093 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:07.895216 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 02:55:07.895307 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:55:07.903553 16714 raft_consensus.cc:1261] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 FOLLOWER]: Refusing update from remote peer fe2684924680452c9bc5ceb47a7c1787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:07.903601 16858 raft_consensus.cc:1261] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Refusing update from remote peer fe2684924680452c9bc5ceb47a7c1787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:07.904377 17074 consensus_queue.cc:1035] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:07.904704 17067 consensus_queue.cc:1035] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:07.915189 17077 mvcc.cc:205] Tried to move back new op lower bound from 7334504480364011520 to 7334504480226488320. Current Snapshot: MvccSnapshot[applied={T|T < 7334504480364011520}]
I20260929 02:55:07.932315 16556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:07.932569 16556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:07.936208 16556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3: Sending DeleteTablet for 3 replicas of tablet 21fb01c83b464f9baf4fee5ca2906973
I20260929 02:55:07.937448 16833 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fb01c83b464f9baf4fee5ca2906973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:07 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:32786
I20260929 02:55:07.937511 16694 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fb01c83b464f9baf4fee5ca2906973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:07 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:48254
I20260929 02:55:07.937707 16972 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fb01c83b464f9baf4fee5ca2906973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:07 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:46528
I20260929 02:55:07.938663 17087 tablet_replica.cc:330] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: stopping tablet replica
I20260929 02:55:07.939069 17087 raft_consensus.cc:2229] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:07.939591 17087 raft_consensus.cc:2258] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:07.940196 16556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:07.940886 16556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:07.941735 17087 ts_tablet_manager.cc:1905] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:07.944031 17088 tablet_replica.cc:330] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: stopping tablet replica
I20260929 02:55:07.944243 17086 tablet_replica.cc:330] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: stopping tablet replica
I20260929 02:55:07.944453 17088 raft_consensus.cc:2229] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:07.944653 17086 raft_consensus.cc:2229] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:07.945039 17088 raft_consensus.cc:2258] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:07.945163 17086 raft_consensus.cc:2258] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:07.947606 17086 ts_tablet_manager.cc:1905] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:07.948571 17087 ts_tablet_manager.cc:1918] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:07.948725 17087 log.cc:1196] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/21fb01c83b464f9baf4fee5ca2906973
I20260929 02:55:07.949242 17087 ts_tablet_manager.cc:1939] T 21fb01c83b464f9baf4fee5ca2906973 P f503956960dc410fa0c8aaf472570b80: Deleting consensus metadata
I20260929 02:55:07.949908 16833 tablet_service.cc:1467] Processing CreateTablet for tablet 0f89b89fe9ef40628b0cfec12de87208 (DEFAULT_TABLE table=test-table [id=4460bb9b3baa4192baef4533d97f9107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:07.950342 16833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f89b89fe9ef40628b0cfec12de87208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:07.951002 16544 catalog_manager.cc:4841] TS f503956960dc410fa0c8aaf472570b80 (127.15.18.66:46009): tablet 21fb01c83b464f9baf4fee5ca2906973 (table test-table [id=1353677845014aa3900167651fbaa7ef]) successfully deleted
I20260929 02:55:07.951719 16972 tablet_service.cc:1467] Processing CreateTablet for tablet 0f89b89fe9ef40628b0cfec12de87208 (DEFAULT_TABLE table=test-table [id=4460bb9b3baa4192baef4533d97f9107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:07.952174 16972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f89b89fe9ef40628b0cfec12de87208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:07.953222 17088 ts_tablet_manager.cc:1905] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:07.955011 16694 tablet_service.cc:1467] Processing CreateTablet for tablet 0f89b89fe9ef40628b0cfec12de87208 (DEFAULT_TABLE table=test-table [id=4460bb9b3baa4192baef4533d97f9107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:07.955102 17086 ts_tablet_manager.cc:1918] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:07.955447 17086 log.cc:1196] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/21fb01c83b464f9baf4fee5ca2906973
I20260929 02:55:07.955536 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f89b89fe9ef40628b0cfec12de87208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:07.959652 17061 tablet_bootstrap.cc:492] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: Bootstrap starting.
I20260929 02:55:07.960054 17062 tablet_bootstrap.cc:492] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: Bootstrap starting.
I20260929 02:55:07.960472 17086 ts_tablet_manager.cc:1939] T 21fb01c83b464f9baf4fee5ca2906973 P 922cf3d5bd73482c8b13e51b22eabd58: Deleting consensus metadata
I20260929 02:55:07.961184 17061 tablet_bootstrap.cc:654] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:07.961562 17063 tablet_bootstrap.cc:492] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: Bootstrap starting.
I20260929 02:55:07.961580 17062 tablet_bootstrap.cc:654] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:07.962692 16542 catalog_manager.cc:4841] TS 922cf3d5bd73482c8b13e51b22eabd58 (127.15.18.65:34351): tablet 21fb01c83b464f9baf4fee5ca2906973 (table test-table [id=1353677845014aa3900167651fbaa7ef]) successfully deleted
I20260929 02:55:07.963078 17063 tablet_bootstrap.cc:654] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:07.972541 17063 tablet_bootstrap.cc:492] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: No bootstrap required, opened a new log
I20260929 02:55:07.972715 17063 ts_tablet_manager.cc:1397] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260929 02:55:07.973599 17063 raft_consensus.cc:348] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:07.973840 17063 raft_consensus.cc:374] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:07.973944 17063 raft_consensus.cc:729] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922cf3d5bd73482c8b13e51b22eabd58, State: Initialized, Role: FOLLOWER
I20260929 02:55:07.972592 17088 ts_tablet_manager.cc:1918] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:07.974217 17088 log.cc:1196] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/21fb01c83b464f9baf4fee5ca2906973
I20260929 02:55:07.975003 17063 consensus_queue.cc:260] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:07.975718 17088 ts_tablet_manager.cc:1939] T 21fb01c83b464f9baf4fee5ca2906973 P fe2684924680452c9bc5ceb47a7c1787: Deleting consensus metadata
I20260929 02:55:07.976076 17062 tablet_bootstrap.cc:492] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: No bootstrap required, opened a new log
I20260929 02:55:07.976231 17062 ts_tablet_manager.cc:1397] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20260929 02:55:07.976311 17063 ts_tablet_manager.cc:1428] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 02:55:07.977023 17062 raft_consensus.cc:348] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:07.977258 17062 raft_consensus.cc:374] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:07.977355 17062 raft_consensus.cc:729] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f503956960dc410fa0c8aaf472570b80, State: Initialized, Role: FOLLOWER
I20260929 02:55:07.977619 17062 consensus_queue.cc:260] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:07.977908 17061 tablet_bootstrap.cc:492] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: No bootstrap required, opened a new log
I20260929 02:55:07.978092 17061 ts_tablet_manager.cc:1397] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.004s
I20260929 02:55:07.978920 17061 raft_consensus.cc:348] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:07.979213 17061 raft_consensus.cc:374] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:07.979360 17061 raft_consensus.cc:729] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2684924680452c9bc5ceb47a7c1787, State: Initialized, Role: FOLLOWER
I20260929 02:55:07.979760 17061 consensus_queue.cc:260] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:07.980419 17061 ts_tablet_manager.cc:1428] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:07.980432 17062 ts_tablet_manager.cc:1428] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:55:07.982016 16541 catalog_manager.cc:4841] TS fe2684924680452c9bc5ceb47a7c1787 (127.15.18.67:41977): tablet 21fb01c83b464f9baf4fee5ca2906973 (table test-table [id=1353677845014aa3900167651fbaa7ef]) successfully deleted
I20260929 02:55:08.169850 17069 raft_consensus.cc:482] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:08.170115 17069 raft_consensus.cc:504] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.172061 17069 leader_election.cc:288] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f503956960dc410fa0c8aaf472570b80 (127.15.18.66:46009), fe2684924680452c9bc5ceb47a7c1787 (127.15.18.67:41977)
W20260929 02:55:08.177112 17043 tablet.cc:2369] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:08.179807 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.172615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:57415 (local address 127.15.18.66:46009)
0929 02:55:08.172790 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.172795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.173355 (+ 560us) server_negotiation.cc:407] Connection header received
0929 02:55:08.173677 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.173684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.173757 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.173838 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:08.174411 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.175476 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.177930 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.178776 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.178920 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.178998 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.179311 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.179441 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:55:08.180238 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.172499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:41047 (local address 127.15.18.67:41977)
0929 02:55:08.172691 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.172698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.173636 (+ 938us) server_negotiation.cc:407] Connection header received
0929 02:55:08.173900 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.173906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.173990 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.174084 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:08.174633 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.175344 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.177908 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.179285 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.179431 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.179568 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.179818 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.179955 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:55:08.180275 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.172280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:41977 (local address 127.15.18.65:41047)
0929 02:55:08.173544 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:08.173557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:08.173730 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:08.174114 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:08.174118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:08.174143 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:08.174501 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.174509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.175461 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.175466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.177740 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.177758 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.179485 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.179490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.179674 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.179704 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:55:08.179793 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 02:55:08.179930 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"mutex_wait_us":458,"thread_start_us":552,"threads_started":1}
I20260929 02:55:08.180872 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.172280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:46009 (local address 127.15.18.65:57415)
0929 02:55:08.173237 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:08.173258 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:08.173497 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:08.173875 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:08.173879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:08.173901 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:08.174216 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.174228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.175606 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.175610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.177777 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.177787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.178965 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.178971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.179116 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.179140 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:55:08.180180 (+ 1040us) client_negotiation.cc:240] Negotiation successful
0929 02:55:08.180571 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":104,"thread_start_us":656,"threads_started":1}
I20260929 02:55:08.180936 16992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f89b89fe9ef40628b0cfec12de87208" candidate_uuid: "922cf3d5bd73482c8b13e51b22eabd58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe2684924680452c9bc5ceb47a7c1787" is_pre_election: true
I20260929 02:55:08.181205 16992 raft_consensus.cc:2454] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922cf3d5bd73482c8b13e51b22eabd58 in term 0.
I20260929 02:55:08.181458 16858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f89b89fe9ef40628b0cfec12de87208" candidate_uuid: "922cf3d5bd73482c8b13e51b22eabd58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f503956960dc410fa0c8aaf472570b80" is_pre_election: true
I20260929 02:55:08.181686 16858 raft_consensus.cc:2454] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 922cf3d5bd73482c8b13e51b22eabd58 in term 0.
I20260929 02:55:08.181960 16644 leader_election.cc:302] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 922cf3d5bd73482c8b13e51b22eabd58, fe2684924680452c9bc5ceb47a7c1787; no voters:
I20260929 02:55:08.182382 17069 raft_consensus.cc:2781] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:08.182495 17069 raft_consensus.cc:482] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:08.182585 17069 raft_consensus.cc:3037] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:08.184854 17069 raft_consensus.cc:504] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.185472 17069 leader_election.cc:288] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [CANDIDATE]: Term 1 election: Requested vote from peers f503956960dc410fa0c8aaf472570b80 (127.15.18.66:46009), fe2684924680452c9bc5ceb47a7c1787 (127.15.18.67:41977)
I20260929 02:55:08.185945 16858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f89b89fe9ef40628b0cfec12de87208" candidate_uuid: "922cf3d5bd73482c8b13e51b22eabd58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f503956960dc410fa0c8aaf472570b80"
I20260929 02:55:08.186152 16858 raft_consensus.cc:3037] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:08.188436 16858 raft_consensus.cc:2454] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922cf3d5bd73482c8b13e51b22eabd58 in term 1.
W20260929 02:55:08.188509 16904 tablet.cc:2369] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:08.188947 16647 leader_election.cc:302] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 922cf3d5bd73482c8b13e51b22eabd58, f503956960dc410fa0c8aaf472570b80; no voters:
I20260929 02:55:08.189234 17069 raft_consensus.cc:2781] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:08.189755 17069 raft_consensus.cc:686] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 LEADER]: Becoming Leader. State: Replica: 922cf3d5bd73482c8b13e51b22eabd58, State: Running, Role: LEADER
I20260929 02:55:08.190169 17069 consensus_queue.cc:237] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.193738 16557 catalog_manager.cc:5495] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 922cf3d5bd73482c8b13e51b22eabd58 (127.15.18.65). New cstate: current_term: 1 leader_uuid: "922cf3d5bd73482c8b13e51b22eabd58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } health_report { overall_health: UNKNOWN } } }
I20260929 02:55:08.195746 16992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f89b89fe9ef40628b0cfec12de87208" candidate_uuid: "922cf3d5bd73482c8b13e51b22eabd58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe2684924680452c9bc5ceb47a7c1787"
I20260929 02:55:08.195967 16992 raft_consensus.cc:3037] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:08.198902 16992 raft_consensus.cc:2454] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 922cf3d5bd73482c8b13e51b22eabd58 in term 1.
W20260929 02:55:08.204002 16765 tablet.cc:2369] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:55:08 dist-test-slave-q5f0 krb5kdc[16512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:08.244062 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.230085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:32979 (local address 127.0.0.1:45828)
0929 02:55:08.230726 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:08.230750 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:08.230928 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:08.231337 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:08.231341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:08.232083 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:08.232319 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.232328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.233404 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.233408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.234352 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.234364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.235045 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.235054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.235211 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.235951 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:08.235969 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:08.238412 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:08.241425 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:08.241439 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:08.241447 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:08.241935 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:08.242280 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:08.242284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:08.242288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:08.242438 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:08.242953 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:08.242961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:08.243135 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 02:55:08.243269 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:55:08.243607 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":309,"threads_started":1}
I20260929 02:55:08.244144 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.230185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.15.18.126:32979)
0929 02:55:08.231054 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.231059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.231078 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:55:08.231120 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.231123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.231172 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.231256 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:08.232481 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.233282 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.234479 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.234874 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.234980 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.238624 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:08.238628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:08.238632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:08.238659 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:08.241273 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.242060 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.242064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.242066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.242184 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.242531 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.242536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.242558 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.242822 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:08.242980 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.243359 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.243754 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":751,"thread_start_us":616,"threads_started":1}
I20260929 02:55:08.249445 16557 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:45828:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:55:08.250483 16557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:08.257485 16694 tablet_service.cc:1467] Processing CreateTablet for tablet 7551bd73acc6473fb64f4957870f7c26 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa54d76667414a3c84b8e6d5447f1048]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:55:08.257484 16833 tablet_service.cc:1467] Processing CreateTablet for tablet 7551bd73acc6473fb64f4957870f7c26 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa54d76667414a3c84b8e6d5447f1048]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:55:08.257612 16972 tablet_service.cc:1467] Processing CreateTablet for tablet 7551bd73acc6473fb64f4957870f7c26 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa54d76667414a3c84b8e6d5447f1048]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:55:08.258052 16833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7551bd73acc6473fb64f4957870f7c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:08.258060 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7551bd73acc6473fb64f4957870f7c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:08.258087 16972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7551bd73acc6473fb64f4957870f7c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:08.265421 17061 tablet_bootstrap.cc:492] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: Bootstrap starting.
I20260929 02:55:08.265534 17062 tablet_bootstrap.cc:492] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: Bootstrap starting.
I20260929 02:55:08.267241 17062 tablet_bootstrap.cc:654] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:08.267261 17061 tablet_bootstrap.cc:654] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:08.268388 17063 tablet_bootstrap.cc:492] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: Bootstrap starting.
I20260929 02:55:08.269780 17062 tablet_bootstrap.cc:492] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: No bootstrap required, opened a new log
I20260929 02:55:08.269948 17062 ts_tablet_manager.cc:1397] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:55:08.270099 17063 tablet_bootstrap.cc:654] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:08.270681 17062 raft_consensus.cc:348] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.270874 17062 raft_consensus.cc:374] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:08.270925 17062 raft_consensus.cc:729] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f503956960dc410fa0c8aaf472570b80, State: Initialized, Role: FOLLOWER
I20260929 02:55:08.271174 17062 consensus_queue.cc:260] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.271761 17062 ts_tablet_manager.cc:1428] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:08.272312 17063 tablet_bootstrap.cc:492] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: No bootstrap required, opened a new log
I20260929 02:55:08.272317 17068 tablet_replica.cc:439] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } }
I20260929 02:55:08.272480 17063 ts_tablet_manager.cc:1397] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 02:55:08.272516 17061 tablet_bootstrap.cc:492] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: No bootstrap required, opened a new log
I20260929 02:55:08.272607 17068 tablet_replica.cc:442] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:08.272681 17061 ts_tablet_manager.cc:1397] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20260929 02:55:08.273438 17063 raft_consensus.cc:348] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.273677 17063 raft_consensus.cc:374] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:08.273766 17063 raft_consensus.cc:729] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922cf3d5bd73482c8b13e51b22eabd58, State: Initialized, Role: FOLLOWER
I20260929 02:55:08.273679 17061 raft_consensus.cc:348] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.273933 17061 raft_consensus.cc:374] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:08.274019 17061 raft_consensus.cc:729] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2684924680452c9bc5ceb47a7c1787, State: Initialized, Role: FOLLOWER
I20260929 02:55:08.274039 17063 consensus_queue.cc:260] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.274470 17069 tablet_replica.cc:439] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } }
I20260929 02:55:08.274775 17069 tablet_replica.cc:442] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:08.275499 17061 consensus_queue.cc:260] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.276027 17067 tablet_replica.cc:439] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } }
I20260929 02:55:08.276285 17067 tablet_replica.cc:442] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:08.276721 17061 ts_tablet_manager.cc:1428] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:55:08.274634 17063 ts_tablet_manager.cc:1428] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:55:08.288868 17068 raft_consensus.cc:482] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:08.289077 17068 raft_consensus.cc:504] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.289695 17068 leader_election.cc:288] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 922cf3d5bd73482c8b13e51b22eabd58 (127.15.18.65:34351), fe2684924680452c9bc5ceb47a7c1787 (127.15.18.67:41977)
I20260929 02:55:08.290278 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7551bd73acc6473fb64f4957870f7c26" candidate_uuid: "f503956960dc410fa0c8aaf472570b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922cf3d5bd73482c8b13e51b22eabd58" is_pre_election: true
I20260929 02:55:08.290357 16992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7551bd73acc6473fb64f4957870f7c26" candidate_uuid: "f503956960dc410fa0c8aaf472570b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe2684924680452c9bc5ceb47a7c1787" is_pre_election: true
I20260929 02:55:08.290575 16714 raft_consensus.cc:2454] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f503956960dc410fa0c8aaf472570b80 in term 0.
I20260929 02:55:08.290623 16992 raft_consensus.cc:2454] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f503956960dc410fa0c8aaf472570b80 in term 0.
I20260929 02:55:08.291049 16785 leader_election.cc:302] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 922cf3d5bd73482c8b13e51b22eabd58, f503956960dc410fa0c8aaf472570b80, fe2684924680452c9bc5ceb47a7c1787; no voters:
I20260929 02:55:08.291334 17068 raft_consensus.cc:2781] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:08.291432 17068 raft_consensus.cc:482] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:08.291558 17068 raft_consensus.cc:3037] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:08.293512 17068 raft_consensus.cc:504] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.294109 17068 leader_election.cc:288] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [CANDIDATE]: Term 1 election: Requested vote from peers 922cf3d5bd73482c8b13e51b22eabd58 (127.15.18.65:34351), fe2684924680452c9bc5ceb47a7c1787 (127.15.18.67:41977)
I20260929 02:55:08.294538 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7551bd73acc6473fb64f4957870f7c26" candidate_uuid: "f503956960dc410fa0c8aaf472570b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "922cf3d5bd73482c8b13e51b22eabd58"
I20260929 02:55:08.294618 16992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7551bd73acc6473fb64f4957870f7c26" candidate_uuid: "f503956960dc410fa0c8aaf472570b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe2684924680452c9bc5ceb47a7c1787"
I20260929 02:55:08.294754 16714 raft_consensus.cc:3037] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:08.294806 16992 raft_consensus.cc:3037] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:08.297730 16992 raft_consensus.cc:2454] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f503956960dc410fa0c8aaf472570b80 in term 1.
I20260929 02:55:08.297730 16714 raft_consensus.cc:2454] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f503956960dc410fa0c8aaf472570b80 in term 1.
I20260929 02:55:08.298172 16784 leader_election.cc:302] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 922cf3d5bd73482c8b13e51b22eabd58, f503956960dc410fa0c8aaf472570b80, fe2684924680452c9bc5ceb47a7c1787; no voters:
I20260929 02:55:08.298468 17068 raft_consensus.cc:2781] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:08.298974 17068 raft_consensus.cc:686] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [term 1 LEADER]: Becoming Leader. State: Replica: f503956960dc410fa0c8aaf472570b80, State: Running, Role: LEADER
I20260929 02:55:08.299503 17068 consensus_queue.cc:237] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } }
I20260929 02:55:08.301462 17106 tablet_replica.cc:439] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: TxnStatusTablet state changed. Reason: New leader f503956960dc410fa0c8aaf472570b80. Latest consensus state: current_term: 1 leader_uuid: "f503956960dc410fa0c8aaf472570b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } }
I20260929 02:55:08.301888 17106 tablet_replica.cc:442] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:55:08.302668 17108 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:55:08.302899 17108 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:55:08.303161 16557 catalog_manager.cc:5495] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 reported cstate change: term changed from 0 to 1, leader changed from <none> to f503956960dc410fa0c8aaf472570b80 (127.15.18.66). New cstate: current_term: 1 leader_uuid: "f503956960dc410fa0c8aaf472570b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } health_report { overall_health: UNKNOWN } } }
I20260929 02:55:08.324522 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.317041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.15.18.66:46009)
0929 02:55:08.317220 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.317226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.317249 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:55:08.317498 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.317504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.317616 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.317798 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:08.318272 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.319553 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.321560 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.322335 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.322499 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.323058 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:08.323847 (+ 789us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:08.324001 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.324125 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.324218 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:55:08.324565 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.316933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:46009 (local address 127.0.0.1:32790)
0929 02:55:08.317147 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:08.317168 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:08.317284 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:08.317832 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:08.317891 (+ 59us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:08.317905 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:08.318109 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.318121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.319700 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.319706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.321384 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.321402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.322545 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.322552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.322746 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.322828 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:08.323976 (+ 1148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:08.324003 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 02:55:08.324090 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 02:55:08.324260 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 02:55:08.329535 16714 raft_consensus.cc:1261] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 FOLLOWER]: Refusing update from remote peer f503956960dc410fa0c8aaf472570b80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:08.329649 16992 raft_consensus.cc:1261] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787 [term 1 FOLLOWER]: Refusing update from remote peer f503956960dc410fa0c8aaf472570b80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:08.330310 17068 consensus_queue.cc:1035] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:08.330916 17106 consensus_queue.cc:1035] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:08.345214 17067 tablet_replica.cc:439] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: TxnStatusTablet state changed. Reason: New leader f503956960dc410fa0c8aaf472570b80. Latest consensus state: current_term: 1 leader_uuid: "f503956960dc410fa0c8aaf472570b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } }
I20260929 02:55:08.345536 17067 tablet_replica.cc:442] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:08.345398 17069 tablet_replica.cc:439] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: TxnStatusTablet state changed. Reason: New leader f503956960dc410fa0c8aaf472570b80. Latest consensus state: current_term: 1 leader_uuid: "f503956960dc410fa0c8aaf472570b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } }
I20260929 02:55:08.345698 17069 tablet_replica.cc:442] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:08.348618 17068 tablet_replica.cc:439] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f503956960dc410fa0c8aaf472570b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } }
I20260929 02:55:08.348933 17068 tablet_replica.cc:442] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:55:08.349395 17079 mvcc.cc:205] Tried to move back new op lower bound from 7334504482109411328 to 7334504481998692352. Current Snapshot: MvccSnapshot[applied={T|T < 7334504482109411328}]
I20260929 02:55:08.350755 17106 tablet_replica.cc:439] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f503956960dc410fa0c8aaf472570b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } }
I20260929 02:55:08.350999 17106 tablet_replica.cc:442] T 7551bd73acc6473fb64f4957870f7c26 P f503956960dc410fa0c8aaf472570b80: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:55:08.352327 17078 mvcc.cc:205] Tried to move back new op lower bound from 7334504482109411328 to 7334504481998692352. Current Snapshot: MvccSnapshot[applied={T|T < 7334504482109411328}]
I20260929 02:55:08.352712 17069 tablet_replica.cc:439] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f503956960dc410fa0c8aaf472570b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } }
I20260929 02:55:08.353032 17069 tablet_replica.cc:442] T 7551bd73acc6473fb64f4957870f7c26 P 922cf3d5bd73482c8b13e51b22eabd58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:08.353024 17067 tablet_replica.cc:439] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f503956960dc410fa0c8aaf472570b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "922cf3d5bd73482c8b13e51b22eabd58" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34351 } } peers { permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 } } peers { permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 } } }
I20260929 02:55:08.353276 17067 tablet_replica.cc:442] T 7551bd73acc6473fb64f4957870f7c26 P fe2684924680452c9bc5ceb47a7c1787: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:08.370988 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.359439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.15.18.65:34351)
0929 02:55:08.363918 (+ 4479us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.363925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.363951 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:55:08.364068 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.364073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.364181 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.364391 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:08.364970 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.366112 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.368195 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.368819 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.368954 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.369509 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:08.370295 (+ 786us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:08.370439 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.370561 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.370650 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4318,"thread_start_us":4182,"threads_started":1}
I20260929 02:55:08.394459 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.382958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:59281 (local address 127.15.18.66:46009)
0929 02:55:08.383133 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.383140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.388209 (+ 5069us) server_negotiation.cc:407] Connection header received
0929 02:55:08.388473 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.388479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.388576 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.388664 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:08.389390 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.390516 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.392314 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.393049 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.393169 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.393690 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:08.393834 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:08.393925 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.394070 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.394160 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:55:08.394552 17116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.382886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:46009 (local address 127.15.18.65:59281)
0929 02:55:08.388057 (+ 5171us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:08.388079 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:08.388273 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:08.388826 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:08.388853 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:08.388868 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:08.389190 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.389201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.390679 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.390684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.392162 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.392179 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.393224 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.393230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.393407 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.393485 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:08.393934 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:08.393944 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:55:08.394018 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:55:08.394163 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4953,"thread_start_us":4812,"threads_started":1}
I20260929 02:55:08.404357 16992 raft_consensus.cc:1261] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787 [term 1 FOLLOWER]: Refusing update from remote peer 922cf3d5bd73482c8b13e51b22eabd58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:08.404417 16858 raft_consensus.cc:1261] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Refusing update from remote peer 922cf3d5bd73482c8b13e51b22eabd58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:08.404992 17094 consensus_queue.cc:1035] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f503956960dc410fa0c8aaf472570b80" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:08.405388 17069 consensus_queue.cc:1035] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe2684924680452c9bc5ceb47a7c1787" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:08.439095 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.432464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:43699 (local address 127.15.18.65:34351)
0929 02:55:08.432634 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.432641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.433254 (+ 613us) server_negotiation.cc:407] Connection header received
0929 02:55:08.433459 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.433465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.433547 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.433631 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:08.434325 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.435180 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.437267 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.437732 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.437840 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.438375 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:08.438509 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:08.438593 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.438732 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.438810 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:55:08.439393 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.432424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:34351 (local address 127.15.18.66:43699)
0929 02:55:08.433151 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:08.433172 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:08.433323 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:08.433785 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:08.433794 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:08.433807 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:08.434168 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.434180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.435312 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.435317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.437120 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:08.437135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.437869 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.437876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:08.438023 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.438091 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:08.438617 (+ 526us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:08.438627 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:55:08.438695 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 02:55:08.438827 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":400,"threads_started":1}
I20260929 02:55:08.480808 16557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:08.481055 16557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:08.484472 16557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f0b13048f364eee80dd40f225f5b0f3: Sending DeleteTablet for 3 replicas of tablet 0f89b89fe9ef40628b0cfec12de87208
I20260929 02:55:08.485471 16694 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f89b89fe9ef40628b0cfec12de87208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:08 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:48254
I20260929 02:55:08.485569 16972 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f89b89fe9ef40628b0cfec12de87208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:08 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:46528
I20260929 02:55:08.486271 16833 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f89b89fe9ef40628b0cfec12de87208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:08 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:32786
I20260929 02:55:08.492245 17127 tablet_replica.cc:330] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: stopping tablet replica
I20260929 02:55:08.492866 17127 raft_consensus.cc:2229] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:08.493435 17127 raft_consensus.cc:2258] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:08.494838 17127 ts_tablet_manager.cc:1905] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:08.508260 17127 ts_tablet_manager.cc:1918] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:08.508996 17127 log.cc:1196] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/0f89b89fe9ef40628b0cfec12de87208
I20260929 02:55:08.510149 17127 ts_tablet_manager.cc:1939] T 0f89b89fe9ef40628b0cfec12de87208 P fe2684924680452c9bc5ceb47a7c1787: Deleting consensus metadata
I20260929 02:55:08.511991 16541 catalog_manager.cc:4841] TS fe2684924680452c9bc5ceb47a7c1787 (127.15.18.67:41977): tablet 0f89b89fe9ef40628b0cfec12de87208 (table test-table [id=4460bb9b3baa4192baef4533d97f9107]) successfully deleted
I20260929 02:55:08.513211 17126 tablet_replica.cc:330] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: stopping tablet replica
I20260929 02:55:08.513437 17126 raft_consensus.cc:2229] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:08.514007 17128 tablet_replica.cc:330] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: stopping tablet replica
I20260929 02:55:08.514326 17128 raft_consensus.cc:2229] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:08.514631 17128 raft_consensus.cc:2258] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:08.516592 17128 ts_tablet_manager.cc:1905] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:08.518719 17126 raft_consensus.cc:2258] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:08.520010 17126 ts_tablet_manager.cc:1905] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:55:08 dist-test-slave-q5f0 krb5kdc[16512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650507, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:08.527359 17128 ts_tablet_manager.cc:1918] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:08.527504 17128 log.cc:1196] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/0f89b89fe9ef40628b0cfec12de87208
I20260929 02:55:08.528040 17128 ts_tablet_manager.cc:1939] T 0f89b89fe9ef40628b0cfec12de87208 P f503956960dc410fa0c8aaf472570b80: Deleting consensus metadata
I20260929 02:55:08.529239 16544 catalog_manager.cc:4841] TS f503956960dc410fa0c8aaf472570b80 (127.15.18.66:46009): tablet 0f89b89fe9ef40628b0cfec12de87208 (table test-table [id=4460bb9b3baa4192baef4533d97f9107]) successfully deleted
I20260929 02:55:08.535871 17126 ts_tablet_manager.cc:1918] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:08.536022 17126 log.cc:1196] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/0f89b89fe9ef40628b0cfec12de87208
I20260929 02:55:08.536047 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.506900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.15.18.65:34351)
0929 02:55:08.507110 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.507117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.514639 (+ 7522us) server_negotiation.cc:407] Connection header received
0929 02:55:08.515047 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.515053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.515194 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.515298 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:08.517550 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.518719 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.521114 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.521829 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.521949 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.527300 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:08.527318 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:08.527339 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:08.527384 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:08.532193 (+ 4809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.533183 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.533198 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.533202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.533390 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.533875 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.533881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.533888 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.534370 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:08.534698 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.535290 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.535539 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 02:55:08.536520 17126 ts_tablet_manager.cc:1939] T 0f89b89fe9ef40628b0cfec12de87208 P 922cf3d5bd73482c8b13e51b22eabd58: Deleting consensus metadata
I20260929 02:55:08.537637 16542 catalog_manager.cc:4841] TS 922cf3d5bd73482c8b13e51b22eabd58 (127.15.18.65:34351): tablet 0f89b89fe9ef40628b0cfec12de87208 (table test-table [id=4460bb9b3baa4192baef4533d97f9107]) successfully deleted
W20260929 02:55:08.538219 16759 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48268
I20260929 02:55:08.559250 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.542363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.15.18.126:32979)
0929 02:55:08.542601 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.542607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.543093 (+ 486us) server_negotiation.cc:407] Connection header received
0929 02:55:08.543339 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.543346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.543442 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.543621 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:08.545369 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.546603 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.548570 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.549305 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.549438 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.552465 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:08.552472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:08.552477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:08.552525 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:08.556147 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.556989 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.556996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.557000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.557169 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.557593 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.557598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.557602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.557997 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:08.558255 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.558613 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.558908 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260929 02:55:08.560114 16557 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45840
I20260929 02:55:08.562109 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 16628
I20260929 02:55:08.579485 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.564762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48284 (local address 127.15.18.65:34351)
0929 02:55:08.565015 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.565021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.565039 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:08.565095 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.565100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.565177 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.565274 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:08.566786 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.568021 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.569697 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.570391 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.570522 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.573216 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:08.573222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:08.573228 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:08.573268 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:08.576769 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.577485 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.577490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.577494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.577656 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.577989 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.577994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.577997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.578335 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:08.578607 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.578856 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.579099 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260929 02:55:08.580132 16759 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48284
W20260929 02:55:08.580718 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 16628: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:55:08.580907 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 16628
I20260929 02:55:08.596104 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 16768
Sep 29 02:55:08 dist-test-slave-q5f0 krb5kdc[16512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650507, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:55:08.611858 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.596917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.15.18.66:46009)
0929 02:55:08.597097 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.597102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.597177 (+ 75us) server_negotiation.cc:407] Connection header received
0929 02:55:08.597365 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.597371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.597451 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.597569 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:08.598748 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.599703 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.601249 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.601728 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.601846 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.605986 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:08.606000 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:08.606016 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:08.606058 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:08.609115 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.609790 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.609797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.609799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.609953 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.610317 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.610321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.610323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.610684 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:08.610999 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.611264 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.611483 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 02:55:08.612486 16898 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32794
W20260929 02:55:08.613135 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 16768: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:55:08.613288 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 16768
I20260929 02:55:08.628427 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 16907
Sep 29 02:55:08 dist-test-slave-q5f0 krb5kdc[16512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650507, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:55:08.643030 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:08.629163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.15.18.67:41977)
0929 02:55:08.629339 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:08.629345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:08.629363 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:08.629562 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:08.629566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:08.629639 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:08.629732 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:08.630889 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.631644 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.632818 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:08.633213 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:08.633317 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:08.637249 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:08.637265 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:08.637281 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:08.637316 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:08.640330 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.641016 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.641027 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.641031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.641159 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:08.641503 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:08.641508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:08.641512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:08.641884 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:08.642197 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:08.642468 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 02:55:08.642687 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 02:55:08.643759 17037 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46546
W20260929 02:55:08.644407 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 16907: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:55:08.644536 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 16907
I20260929 02:55:08.660945 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 16525
I20260929 02:55:08.662089 16597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:08.776539 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 16525
2026-09-29T02:55:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4274 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:08 dist-test-slave-q5f0 krb5kdc[17137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:08 dist-test-slave-q5f0 krb5kdc[17137](info): set up 2 sockets
Sep 29 02:55:08 dist-test-slave-q5f0 krb5kdc[17137](info): commencing operation
krb5kdc: starting...
W20260929 02:55:09.779846 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:55:09 dist-test-slave-q5f0 krb5kdc[17137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:55:09.960690 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:34323
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:34323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:10.279173 17150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:10.279471 17150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:10.279614 17150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:10.286820 17150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:10.286936 17150 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:55:10.286986 17150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:10.287070 17150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:10.287137 17150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:10.287220 17150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:10.297339 17150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:34323
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:34323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:10.298636 17150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:10.300045 17150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:10.311501 17155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:10.311576 17156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:10.311579 17158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:10.312649 17150 server_base.cc:1034] running on GCE node
I20260929 02:55:10.313400 17150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:10.314818 17150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:10.316053 17150 hybrid_clock.cc:648] HybridClock initialized: now 1790650510315984 us; error 44 us; skew 500 ppm
I20260929 02:55:10.316413 17150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:10 dist-test-slave-q5f0 krb5kdc[17137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650510, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:10.319612 17150 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:55:10.321028 17150 webserver.cc:460] Webserver started at http://127.15.18.126:40647/ using document root <none> and password file <none>
I20260929 02:55:10.321724 17150 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:10.321820 17150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:10.322114 17150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:10.325192 17150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "01b90c2a475a4e6298f6cc173e856d86"
format_stamp: "Formatted at 2026-09-29 02:55:10 on dist-test-slave-q5f0"
server_key: "4711545d249d1a27229b863ea8afbbf9"
server_key_iv: "d1c1718f75610fa471cbaad18fb52bcc"
server_key_version: "encryptionkey@0"
I20260929 02:55:10.326011 17150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "01b90c2a475a4e6298f6cc173e856d86"
format_stamp: "Formatted at 2026-09-29 02:55:10 on dist-test-slave-q5f0"
server_key: "4711545d249d1a27229b863ea8afbbf9"
server_key_iv: "d1c1718f75610fa471cbaad18fb52bcc"
server_key_version: "encryptionkey@0"
I20260929 02:55:10.331423 17150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:55:10.335026 17165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:10.336479 17150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:55:10.336655 17150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "01b90c2a475a4e6298f6cc173e856d86"
format_stamp: "Formatted at 2026-09-29 02:55:10 on dist-test-slave-q5f0"
server_key: "4711545d249d1a27229b863ea8afbbf9"
server_key_iv: "d1c1718f75610fa471cbaad18fb52bcc"
server_key_version: "encryptionkey@0"
I20260929 02:55:10.336855 17150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:10.365197 17150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:10.369575 17150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:10.369823 17150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:10.389894 17150 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:34323
I20260929 02:55:10.389918 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:34323 every 8 connection(s)
I20260929 02:55:10.391209 17150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:10.395071 17227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:10.398656 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 17150
I20260929 02:55:10.398859 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:10.399660 15433 external_mini_cluster.cc:1428] Setting key 6d3b7e770eb7300d08b1ac14828591d3
I20260929 02:55:10.407999 17227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86: Bootstrap starting.
I20260929 02:55:10.411916 17227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:10.413036 17227 log.cc:824] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:55:10 dist-test-slave-q5f0 krb5kdc[17137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:10.416780 17227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86: No bootstrap required, opened a new log
I20260929 02:55:10.423924 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:10.401795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.15.18.126:34323)
0929 02:55:10.402538 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:10.402550 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:10.402593 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:55:10.404272 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:10.404303 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:10.404793 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:10.405205 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:10.406732 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:10.408207 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:10.409530 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:10.410329 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:10.410530 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:10.414908 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:10.414921 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:10.414944 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:10.414999 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:10.418993 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:10.419835 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:10.419845 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:10.419858 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:10.420087 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:10.420451 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:10.420456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:10.420459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:10.421214 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:10.421508 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:10.421933 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 02:55:10.422421 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":350,"threads_started":1}
I20260929 02:55:10.426797 17227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b90c2a475a4e6298f6cc173e856d86" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34323 } }
I20260929 02:55:10.427188 17227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:10.427289 17227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b90c2a475a4e6298f6cc173e856d86, State: Initialized, Role: FOLLOWER
I20260929 02:55:10.427870 17227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b90c2a475a4e6298f6cc173e856d86" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34323 } }
I20260929 02:55:10.428097 17227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:10.428215 17227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:10.428362 17227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:10.430505 17227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b90c2a475a4e6298f6cc173e856d86" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34323 } }
I20260929 02:55:10.430979 17227 leader_election.cc:302] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01b90c2a475a4e6298f6cc173e856d86; no voters:
I20260929 02:55:10.431586 17227 leader_election.cc:288] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:10.431694 17232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:10.432288 17232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [term 1 LEADER]: Becoming Leader. State: Replica: 01b90c2a475a4e6298f6cc173e856d86, State: Running, Role: LEADER
I20260929 02:55:10.432889 17232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b90c2a475a4e6298f6cc173e856d86" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34323 } }
I20260929 02:55:10.433697 17227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:10.435663 17234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b90c2a475a4e6298f6cc173e856d86. Latest consensus state: current_term: 1 leader_uuid: "01b90c2a475a4e6298f6cc173e856d86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b90c2a475a4e6298f6cc173e856d86" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34323 } } }
I20260929 02:55:10.436156 17234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:10.435789 17233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01b90c2a475a4e6298f6cc173e856d86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b90c2a475a4e6298f6cc173e856d86" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34323 } } }
I20260929 02:55:10.436650 17233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:10.437794 17241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:10.442746 17241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:10.451121 17241 catalog_manager.cc:1347] Generated new cluster ID: 80be3e2baa694bca9adf28ade8d33aa0
I20260929 02:55:10.451231 17241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:10.468822 17241 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:10.470887 17241 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:10.480330 17241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86: Generated new TSK 0
I20260929 02:55:10.481251 17241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:55:10.535626 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34323
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:10.846244 17253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:10.846521 17253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:10.846668 17253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:10.853842 17253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:10.853950 17253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:10.854079 17253 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:55:10.854143 17253 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:55:10.854246 17253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:10.863988 17253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.18.126:34323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:10.865275 17253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:10.866550 17253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:10.879359 17258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:10.879420 17261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:10.879453 17259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:10.879774 17253 server_base.cc:1034] running on GCE node
I20260929 02:55:10.880625 17253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:10.881546 17253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:10.882750 17253 hybrid_clock.cc:648] HybridClock initialized: now 1790650510882674 us; error 57 us; skew 500 ppm
I20260929 02:55:10.883083 17253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:10 dist-test-slave-q5f0 krb5kdc[17137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650510, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:10.886199 17253 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:55:10.887636 17253 webserver.cc:460] Webserver started at http://127.15.18.65:45575/ using document root <none> and password file <none>
I20260929 02:55:10.888288 17253 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:10.888386 17253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:10.888676 17253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:10.891547 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "060d98b7b1844179b502b5ed460817d8"
format_stamp: "Formatted at 2026-09-29 02:55:10 on dist-test-slave-q5f0"
server_key: "0b4186a072cbfeacb3e752f5c787f8ae"
server_key_iv: "96f07bdf66691f9959682b4074ae5b8b"
server_key_version: "encryptionkey@0"
I20260929 02:55:10.892279 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "060d98b7b1844179b502b5ed460817d8"
format_stamp: "Formatted at 2026-09-29 02:55:10 on dist-test-slave-q5f0"
server_key: "0b4186a072cbfeacb3e752f5c787f8ae"
server_key_iv: "96f07bdf66691f9959682b4074ae5b8b"
server_key_version: "encryptionkey@0"
I20260929 02:55:10.897272 17253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:55:10.900597 17268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:10.901908 17253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:55:10.902053 17253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "060d98b7b1844179b502b5ed460817d8"
format_stamp: "Formatted at 2026-09-29 02:55:10 on dist-test-slave-q5f0"
server_key: "0b4186a072cbfeacb3e752f5c787f8ae"
server_key_iv: "96f07bdf66691f9959682b4074ae5b8b"
server_key_version: "encryptionkey@0"
I20260929 02:55:10.902227 17253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:10.927012 17253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:10.931223 17253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:10.931501 17253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:10.934147 17253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:10.934262 17253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:10.934374 17253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:10.934458 17253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:55:10 dist-test-slave-q5f0 krb5kdc[17137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650510, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:10.955641 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:10.934792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:58171 (local address 127.15.18.126:34323)
0929 02:55:10.935256 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:10.935262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:10.936625 (+ 1363us) server_negotiation.cc:407] Connection header received
0929 02:55:10.938885 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:10.938890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:10.938973 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:10.939067 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:10.941246 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:10.941940 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:10.944258 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:10.944675 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:10.944783 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:10.949518 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:10.949524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:10.949527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:10.949574 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:10.952311 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:10.953142 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:10.953150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:10.953155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:10.953320 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:10.953722 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:10.953727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:10.953731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:10.954133 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:10.954344 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:10.954954 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 02:55:10.955204 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260929 02:55:10.956512 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:10.935631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34323 (local address 127.15.18.65:58171)
0929 02:55:10.936478 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:10.936524 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:10.938619 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:10.939328 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:10.939336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:10.940110 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:10.941037 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:10.941068 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:10.942091 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:10.942096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:10.944123 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:10.944135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:10.944894 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:10.944901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:10.945125 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:10.945808 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:10.945833 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:10.949215 (+ 3382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:10.952501 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:10.952511 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:10.952533 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:10.952996 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:10.953419 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:10.953423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:10.953425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:10.953618 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:10.954245 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:10.954251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:10.954601 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 02:55:10.954821 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 02:55:10.955204 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":371,"threads_started":1}
I20260929 02:55:10.974129 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:10.961899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:57421 (local address 127.15.18.126:34323)
0929 02:55:10.962056 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:10.962060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:10.962652 (+ 592us) server_negotiation.cc:407] Connection header received
0929 02:55:10.962803 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:10.962806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:10.962862 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:10.962938 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:10.963877 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:10.964613 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:10.965859 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:10.966285 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:10.966388 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:10.968151 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:10.968159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:10.968165 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:10.968203 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:10.971339 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:10.972158 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:10.972164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:10.972168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:10.972321 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:10.972713 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:10.972717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:10.972720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:10.973070 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:10.973200 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:10.973598 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 02:55:10.973797 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 02:55:10.974216 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:10.962007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34323 (local address 127.15.18.65:57421)
0929 02:55:10.962563 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:10.962577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:10.962695 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:10.962974 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:10.962979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:10.963394 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:10.963748 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:10.963759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:10.964745 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:10.964750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:10.965734 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:10.965745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:10.966448 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:10.966455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:10.966633 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:10.967170 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:10.967185 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:10.967962 (+ 777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:10.971489 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:10.971494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:10.971498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:10.972026 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:10.972458 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:10.972462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:10.972466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:10.972597 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:10.973181 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:10.973185 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:10.973362 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 02:55:10.973474 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 02:55:10.973782 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":297,"threads_started":1}
I20260929 02:55:10.990283 17253 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:46685
I20260929 02:55:10.990335 17388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:46685 every 8 connection(s)
I20260929 02:55:10.991492 17253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:10.993520 17389 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34323
I20260929 02:55:10.993707 17389 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:10.994402 17389 heartbeater.cc:511] Master 127.15.18.126:34323 requested a full tablet report, sending...
I20260929 02:55:10.995085 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 17253
I20260929 02:55:10.995270 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:10.996027 15433 external_mini_cluster.cc:1428] Setting key 216bac8a58e1d48699cd78dfedadd284
I20260929 02:55:10.996717 17182 ts_manager.cc:194] Registered new tserver with Master: 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685)
I20260929 02:55:10.998467 17182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:58171
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:55:11.035480 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34323
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:11.346711 17393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:11.346989 17393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:11.347119 17393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:11.354554 17393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:11.354658 17393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:11.354784 17393 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:55:11.354856 17393 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:55:11.354956 17393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:11.364909 17393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.18.126:34323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:11.366165 17393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:11.367452 17393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:11.380048 17401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:11.380070 17399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:11.380017 17398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:11.380816 17393 server_base.cc:1034] running on GCE node
I20260929 02:55:11.381464 17393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:11.382470 17393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:11.383643 17393 hybrid_clock.cc:648] HybridClock initialized: now 1790650511383589 us; error 49 us; skew 500 ppm
I20260929 02:55:11.384001 17393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:11 dist-test-slave-q5f0 krb5kdc[17137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:11.387135 17393 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:55:11.388684 17393 webserver.cc:460] Webserver started at http://127.15.18.66:41039/ using document root <none> and password file <none>
I20260929 02:55:11.389421 17393 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:11.389545 17393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:11.389851 17393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:11.392750 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "2674752f524e4805ac74e3cce351a8d9"
format_stamp: "Formatted at 2026-09-29 02:55:11 on dist-test-slave-q5f0"
server_key: "32cf014395d6e8585a07bc1c0d64d751"
server_key_iv: "26fad30769fb7634cec8211409041f22"
server_key_version: "encryptionkey@0"
I20260929 02:55:11.393469 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "2674752f524e4805ac74e3cce351a8d9"
format_stamp: "Formatted at 2026-09-29 02:55:11 on dist-test-slave-q5f0"
server_key: "32cf014395d6e8585a07bc1c0d64d751"
server_key_iv: "26fad30769fb7634cec8211409041f22"
server_key_version: "encryptionkey@0"
I20260929 02:55:11.398690 17393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:11.402433 17408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:11.403915 17393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:55:11.404090 17393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "2674752f524e4805ac74e3cce351a8d9"
format_stamp: "Formatted at 2026-09-29 02:55:11 on dist-test-slave-q5f0"
server_key: "32cf014395d6e8585a07bc1c0d64d751"
server_key_iv: "26fad30769fb7634cec8211409041f22"
server_key_version: "encryptionkey@0"
I20260929 02:55:11.404263 17393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:11.426682 17393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:11.432060 17393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:11.432389 17393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:11.436000 17393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:11.436131 17393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:11.436293 17393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:11.436414 17393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:55:11 dist-test-slave-q5f0 krb5kdc[17137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:11.458299 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:11.437772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:52305 (local address 127.15.18.126:34323)
0929 02:55:11.437952 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:11.437958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:11.438284 (+ 326us) server_negotiation.cc:407] Connection header received
0929 02:55:11.440592 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:11.440598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:11.440725 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:11.440853 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:11.442719 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.444006 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.447088 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.447631 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.447782 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:11.451502 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:11.451530 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:11.451535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:11.451575 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:11.454415 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:11.455321 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:11.455326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:11.455329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:11.455503 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:11.456133 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:11.456138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:11.456142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:11.456553 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:11.456697 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:11.457732 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0929 02:55:11.457964 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:55:11.459456 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:11.437162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34323 (local address 127.15.18.66:52305)
0929 02:55:11.438115 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:11.438170 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:11.440252 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:11.441077 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:11.441087 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:11.441746 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:11.442495 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:11.442521 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.444468 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.444475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:11.446888 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:11.446909 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.448122 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.448128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:11.448324 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:11.449067 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:11.449093 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:11.451275 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:11.454546 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:11.454558 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:11.454580 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:11.455185 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:11.455696 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:11.455702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:11.455705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:11.456002 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:11.456702 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:11.456711 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:11.457248 (+ 537us) client_negotiation.cc:769] Sending connection context
0929 02:55:11.457647 (+ 399us) client_negotiation.cc:240] Negotiation successful
0929 02:55:11.458254 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":382,"threads_started":1}
I20260929 02:55:11.482273 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:11.466729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:51843 (local address 127.15.18.126:34323)
0929 02:55:11.466929 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:11.466936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:11.468228 (+ 1292us) server_negotiation.cc:407] Connection header received
0929 02:55:11.468356 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:11.468362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:11.468462 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:11.468583 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:11.469506 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.470757 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.472430 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.472958 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.473070 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:11.475605 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:11.475610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:11.475613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:11.475644 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:11.478758 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:11.479773 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:11.479779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:11.479783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:11.479962 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:11.480486 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:11.480492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:11.480495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:11.480974 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:11.481108 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:11.481673 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 02:55:11.481944 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:55:11.482475 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:11.466190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34323 (local address 127.15.18.66:51843)
0929 02:55:11.468109 (+ 1919us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:11.468137 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:11.468257 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:11.468636 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:11.468640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:11.469095 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:11.469371 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:11.469380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.470906 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.470910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:11.472288 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:11.472300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.473156 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.473162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:11.473412 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:11.474110 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:11.474135 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:11.475382 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:11.478956 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:11.478961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:11.478965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:11.479632 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:11.480099 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:11.480106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:11.480110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:11.480284 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:11.481088 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:11.481093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:11.481324 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 02:55:11.481525 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 02:55:11.481956 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1779,"thread_start_us":1515,"threads_started":1}
I20260929 02:55:11.495078 17393 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:44873
I20260929 02:55:11.495111 17527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:44873 every 8 connection(s)
I20260929 02:55:11.496376 17393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:11.498481 17528 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34323
I20260929 02:55:11.498724 17528 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:11.499346 17528 heartbeater.cc:511] Master 127.15.18.126:34323 requested a full tablet report, sending...
I20260929 02:55:11.500746 17182 ts_manager.cc:194] Registered new tserver with Master: 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873)
I20260929 02:55:11.501540 17182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:52305
I20260929 02:55:11.503372 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 17393
I20260929 02:55:11.503556 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:11.503979 15433 external_mini_cluster.cc:1428] Setting key 18e52b69bffcc272702d9636274efd7b
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:55:11.543071 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34323
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:11.862038 17532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:11.862355 17532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:11.862517 17532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:11.869823 17532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:11.869941 17532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:11.870026 17532 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:55:11.870076 17532 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:55:11.870144 17532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:11.880203 17532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.18.126:34323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:11.881498 17532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:11.882787 17532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:11.896149 17537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:11.896235 17538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:11.896260 17540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:11.897591 17532 server_base.cc:1034] running on GCE node
I20260929 02:55:11.898185 17532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:11.899127 17532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:11.900315 17532 hybrid_clock.cc:648] HybridClock initialized: now 1790650511900274 us; error 35 us; skew 500 ppm
I20260929 02:55:11.900601 17532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:11 dist-test-slave-q5f0 krb5kdc[17137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:11.903816 17532 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:55:11.905388 17532 webserver.cc:460] Webserver started at http://127.15.18.67:44503/ using document root <none> and password file <none>
I20260929 02:55:11.906149 17532 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:11.906270 17532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:11.906586 17532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:11.909808 17532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "a62c56ed5cf5477b8c79073ab328546c"
format_stamp: "Formatted at 2026-09-29 02:55:11 on dist-test-slave-q5f0"
server_key: "1d1571b1ca97bda5019cd003ca934044"
server_key_iv: "ea4ee9f45207ed1daa4f22e5468eb7c2"
server_key_version: "encryptionkey@0"
I20260929 02:55:11.910542 17532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "a62c56ed5cf5477b8c79073ab328546c"
format_stamp: "Formatted at 2026-09-29 02:55:11 on dist-test-slave-q5f0"
server_key: "1d1571b1ca97bda5019cd003ca934044"
server_key_iv: "ea4ee9f45207ed1daa4f22e5468eb7c2"
server_key_version: "encryptionkey@0"
I20260929 02:55:11.915922 17532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:55:11.919394 17547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:11.920840 17532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:11.921013 17532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "a62c56ed5cf5477b8c79073ab328546c"
format_stamp: "Formatted at 2026-09-29 02:55:11 on dist-test-slave-q5f0"
server_key: "1d1571b1ca97bda5019cd003ca934044"
server_key_iv: "ea4ee9f45207ed1daa4f22e5468eb7c2"
server_key_version: "encryptionkey@0"
I20260929 02:55:11.921218 17532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:11.940119 17532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:11.944649 17532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:11.944933 17532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:11.947654 17532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:11.947757 17532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:11.947871 17532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:11.947960 17532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:55:11 dist-test-slave-q5f0 krb5kdc[17137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:11.970968 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:11.948472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:42553 (local address 127.15.18.126:34323)
0929 02:55:11.948661 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:11.948665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:11.950282 (+ 1617us) server_negotiation.cc:407] Connection header received
0929 02:55:11.952539 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:11.952546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:11.952630 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:11.952727 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:11.955038 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.956072 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.958560 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.959160 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.959294 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:11.964087 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:11.964092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:11.964096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:11.964129 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:11.967569 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:11.968536 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:11.968542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:11.968546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:11.968686 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:11.969120 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:11.969126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:11.969130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:11.969409 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:11.969551 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:11.970374 (+ 823us) server_negotiation.cc:299] Negotiation successful
0929 02:55:11.970624 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:55:11.972275 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:11.949161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34323 (local address 127.15.18.67:42553)
0929 02:55:11.950120 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:11.950161 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:11.952166 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:11.953195 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:11.953203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:11.953909 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:11.954847 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:11.954873 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.956250 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.956256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:11.958403 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:11.958424 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.959450 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.959463 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:11.959829 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:11.960643 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:11.960665 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:11.963813 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:11.967731 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:11.967743 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:11.967767 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:11.968414 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:11.968788 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:11.968793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:11.968797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:11.969016 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:11.969530 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:11.969538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:11.969962 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 02:55:11.970251 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 02:55:11.970790 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":475,"threads_started":1}
I20260929 02:55:11.993088 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:11.978074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:37635 (local address 127.15.18.126:34323)
0929 02:55:11.978249 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:11.978256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:11.978868 (+ 612us) server_negotiation.cc:407] Connection header received
0929 02:55:11.979062 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:11.979068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:11.979150 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:11.979260 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:11.980624 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.981604 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.983427 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.983943 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.984075 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:11.986459 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:11.986466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:11.986471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:11.986510 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:11.989832 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:11.990757 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:11.990763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:11.990767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:11.990931 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:11.991318 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:11.991324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:11.991327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:11.991715 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:11.991855 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:11.992442 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 02:55:11.992707 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:55:11.993105 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:11.978100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34323 (local address 127.15.18.67:37635)
0929 02:55:11.978736 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:11.978758 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:11.978925 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:11.979317 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:11.979321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:11.979991 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:11.980416 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:11.980437 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.981775 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.981781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:11.983291 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:11.983304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:11.984136 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:11.984143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:11.984302 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:11.984965 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:11.984989 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:11.986237 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:11.989991 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:11.989998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:11.990003 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:11.990600 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:11.991042 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:11.991047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:11.991051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:11.991208 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:11.991922 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:11.991927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:11.992138 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 02:55:11.992291 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 02:55:11.992648 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":319,"threads_started":1}
I20260929 02:55:12.002022 17389 heartbeater.cc:503] Master 127.15.18.126:34323 was elected leader, sending a full tablet report...
I20260929 02:55:12.002364 17532 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:46137
I20260929 02:55:12.002381 17666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:46137 every 8 connection(s)
I20260929 02:55:12.003933 17532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:12.005918 17667 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34323
I20260929 02:55:12.006137 17667 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:12.006814 17667 heartbeater.cc:511] Master 127.15.18.126:34323 requested a full tablet report, sending...
I20260929 02:55:12.008185 17182 ts_manager.cc:194] Registered new tserver with Master: a62c56ed5cf5477b8c79073ab328546c (127.15.18.67:46137)
I20260929 02:55:12.009117 17182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:42553
I20260929 02:55:12.011087 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 17532
I20260929 02:55:12.011279 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:12.011900 15433 external_mini_cluster.cc:1428] Setting key 373f5b9be0bd978f2bb6fa29e0b96a6e
I20260929 02:55:12.015144 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:55:12 dist-test-slave-q5f0 krb5kdc[17137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:55:12 dist-test-slave-q5f0 krb5kdc[17137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:12.060030 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.046536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.15.18.126:34323)
0929 02:55:12.046773 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:12.046781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:12.046978 (+ 197us) server_negotiation.cc:407] Connection header received
0929 02:55:12.047111 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:12.047116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:12.047181 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:12.047268 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:12.048632 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.049487 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.050727 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.051206 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.051313 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.055100 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:12.055106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:12.055109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:12.055148 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:12.057551 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:12.058212 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:12.058217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:12.058219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:12.058353 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:12.058670 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:12.058674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:12.058679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:12.059023 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:12.059209 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:12.059455 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 02:55:12.059661 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 02:55:12.064337 17182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:12.071270 17182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:12.103974 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.083956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:46685 (local address 127.0.0.1:33126)
0929 02:55:12.095773 (+ 11817us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:12.095824 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:12.096011 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:12.096826 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:12.096833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:12.096870 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:12.097325 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.097339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.099204 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.099212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.101404 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.101419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.102976 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.102982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.103259 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.103344 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 02:55:12.103468 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 02:55:12.103660 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11499,"mutex_wait_us":38,"thread_start_us":12418,"threads_started":1}
I20260929 02:55:12.104758 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.084186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.15.18.65:46685)
0929 02:55:12.095560 (+ 11374us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:12.095570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:12.096133 (+ 563us) server_negotiation.cc:407] Connection header received
0929 02:55:12.096243 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:12.096252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:12.096532 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:12.096739 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:12.097542 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.099009 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.101607 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.102759 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.102897 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.103133 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:12.104391 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0929 02:55:12.104536 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11161,"thread_start_us":11018,"threads_started":1}
I20260929 02:55:12.107116 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.084549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46137 (local address 127.0.0.1:53964)
0929 02:55:12.096506 (+ 11957us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:12.096526 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:12.096636 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:12.098738 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:12.098744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:12.098776 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:12.099288 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.099305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.101348 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.101354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.103342 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.103355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.105172 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.105178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.105332 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.105362 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:55:12.105443 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:55:12.105628 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11842,"mutex_wait_us":67,"thread_start_us":14567,"threads_started":1}
I20260929 02:55:12.107290 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.085445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.15.18.66:44873)
0929 02:55:12.097654 (+ 12209us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:12.097662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:12.099593 (+ 1931us) server_negotiation.cc:407] Connection header received
0929 02:55:12.099878 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:12.099888 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:12.100136 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:12.100326 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:12.100955 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.102333 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.104758 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.106075 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.106196 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.106382 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:12.106884 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 02:55:12.107043 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12035,"thread_start_us":11890,"threads_started":1}
I20260929 02:55:12.107586 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.084746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:44873 (local address 127.0.0.1:40346)
0929 02:55:12.099447 (+ 14701us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:12.099464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:12.099694 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:12.100343 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:12.100348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:12.100373 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:12.100781 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.100792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.102478 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.102484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.104593 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.104608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.106277 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.106284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.106450 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.106472 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:55:12.106567 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:55:12.106663 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14572,"mutex_wait_us":41,"thread_start_us":10697,"threads_started":1}
I20260929 02:55:12.107779 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.084702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.15.18.67:46137)
0929 02:55:12.098133 (+ 13431us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:12.098141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:12.098173 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:55:12.098260 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:12.098267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:12.098510 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:12.098707 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:12.099567 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.101183 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.103543 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.104989 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.105112 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.105348 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:12.105483 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 02:55:12.105647 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13262,"thread_start_us":14665,"threads_started":1}
I20260929 02:55:12.108992 17319 tablet_service.cc:1467] Processing CreateTablet for tablet 0eeea682dd124affbf80622f73e634eb (DEFAULT_TABLE table=test-table [id=58b9d5c9ce8644cf81cbc607d9e63929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:12.110553 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eeea682dd124affbf80622f73e634eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:12.111074 17597 tablet_service.cc:1467] Processing CreateTablet for tablet 0eeea682dd124affbf80622f73e634eb (DEFAULT_TABLE table=test-table [id=58b9d5c9ce8644cf81cbc607d9e63929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:12.111228 17462 tablet_service.cc:1467] Processing CreateTablet for tablet 0eeea682dd124affbf80622f73e634eb (DEFAULT_TABLE table=test-table [id=58b9d5c9ce8644cf81cbc607d9e63929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:12.112427 17597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eeea682dd124affbf80622f73e634eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:12.112586 17462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eeea682dd124affbf80622f73e634eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:12.123179 17686 tablet_bootstrap.cc:492] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: Bootstrap starting.
I20260929 02:55:12.125298 17687 tablet_bootstrap.cc:492] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: Bootstrap starting.
I20260929 02:55:12.126891 17686 tablet_bootstrap.cc:654] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:12.127136 17688 tablet_bootstrap.cc:492] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: Bootstrap starting.
I20260929 02:55:12.128263 17686 log.cc:824] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:12.129498 17687 tablet_bootstrap.cc:654] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:12.130244 17688 tablet_bootstrap.cc:654] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:12.130646 17687 log.cc:824] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:12.131223 17688 log.cc:824] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:12.131968 17686 tablet_bootstrap.cc:492] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: No bootstrap required, opened a new log
I20260929 02:55:12.132263 17686 ts_tablet_manager.cc:1397] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 02:55:12.133222 17687 tablet_bootstrap.cc:492] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: No bootstrap required, opened a new log
I20260929 02:55:12.133481 17687 ts_tablet_manager.cc:1397] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:55:12.137383 17688 tablet_bootstrap.cc:492] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: No bootstrap required, opened a new log
I20260929 02:55:12.137669 17688 ts_tablet_manager.cc:1397] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260929 02:55:12.139752 17686 raft_consensus.cc:348] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.140195 17686 raft_consensus.cc:374] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:12.140295 17686 raft_consensus.cc:729] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62c56ed5cf5477b8c79073ab328546c, State: Initialized, Role: FOLLOWER
I20260929 02:55:12.140889 17686 consensus_queue.cc:260] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.142037 17667 heartbeater.cc:503] Master 127.15.18.126:34323 was elected leader, sending a full tablet report...
I20260929 02:55:12.142572 17686 ts_tablet_manager.cc:1428] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 02:55:12.143669 17687 raft_consensus.cc:348] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.144102 17687 raft_consensus.cc:374] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:12.144199 17687 raft_consensus.cc:729] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060d98b7b1844179b502b5ed460817d8, State: Initialized, Role: FOLLOWER
I20260929 02:55:12.144804 17687 consensus_queue.cc:260] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.146497 17687 ts_tablet_manager.cc:1428] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 02:55:12.147135 17688 raft_consensus.cc:348] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.147581 17688 raft_consensus.cc:374] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:12.147686 17688 raft_consensus.cc:729] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2674752f524e4805ac74e3cce351a8d9, State: Initialized, Role: FOLLOWER
I20260929 02:55:12.148237 17688 consensus_queue.cc:260] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.149348 17528 heartbeater.cc:503] Master 127.15.18.126:34323 was elected leader, sending a full tablet report...
I20260929 02:55:12.149930 17688 ts_tablet_manager.cc:1428] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 02:55:12.242642 17692 raft_consensus.cc:482] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:12.243077 17692 raft_consensus.cc:504] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
W20260929 02:55:12.243846 17390 tablet.cc:2369] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:12.245102 17692 leader_election.cc:288] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685), 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873)
W20260929 02:55:12.248229 17529 tablet.cc:2369] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:12.251060 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.245400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:52561 (local address 127.15.18.65:46685)
0929 02:55:12.245594 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:12.245599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:12.245853 (+ 254us) server_negotiation.cc:407] Connection header received
0929 02:55:12.246044 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:12.246049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:12.246115 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:12.246187 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:12.246588 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.247286 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.249311 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.250191 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.250346 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.250420 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:12.250696 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 02:55:12.250823 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:55:12.251091 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.245351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:46685 (local address 127.15.18.67:52561)
0929 02:55:12.245756 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:12.245771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:12.245904 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:12.246229 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:12.246234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:12.246260 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:12.246463 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.246471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.247423 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.247427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.249161 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.249173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.250346 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.250353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.250547 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.250572 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:55:12.250663 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:55:12.250800 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":3,"spinlock_wait_cycles":19200}
I20260929 02:55:12.251293 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.245521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:46389 (local address 127.15.18.66:44873)
0929 02:55:12.245705 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:12.245710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:12.246299 (+ 589us) server_negotiation.cc:407] Connection header received
0929 02:55:12.246478 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:12.246484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:12.246545 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:12.246615 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:12.247103 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.247883 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.249616 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.250438 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.250589 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.250661 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:12.250926 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 02:55:12.251055 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:55:12.251657 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.245515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:44873 (local address 127.15.18.67:46389)
0929 02:55:12.246195 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:12.246213 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:12.246352 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:12.246645 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:12.246649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:12.246672 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:12.246968 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.246976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.248006 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.248011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.249466 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.249480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.250650 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.250656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.250787 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.250811 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:55:12.250891 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 02:55:12.251039 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":31,"thread_start_us":373,"threads_started":1}
I20260929 02:55:12.252074 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eeea682dd124affbf80622f73e634eb" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060d98b7b1844179b502b5ed460817d8" is_pre_election: true
I20260929 02:55:12.252548 17344 raft_consensus.cc:2454] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 0.
I20260929 02:55:12.252554 17483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eeea682dd124affbf80622f73e634eb" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2674752f524e4805ac74e3cce351a8d9" is_pre_election: true
I20260929 02:55:12.253014 17483 raft_consensus.cc:2454] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 0.
I20260929 02:55:12.253167 17549 leader_election.cc:302] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060d98b7b1844179b502b5ed460817d8, a62c56ed5cf5477b8c79073ab328546c; no voters:
I20260929 02:55:12.253564 17692 raft_consensus.cc:2781] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:12.253695 17692 raft_consensus.cc:482] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:12.253791 17692 raft_consensus.cc:3037] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:55:12.255375 17668 tablet.cc:2369] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:12.256196 17692 raft_consensus.cc:504] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.256832 17692 leader_election.cc:288] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 election: Requested vote from peers 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685), 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873)
I20260929 02:55:12.257416 17483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eeea682dd124affbf80622f73e634eb" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2674752f524e4805ac74e3cce351a8d9"
I20260929 02:55:12.257486 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eeea682dd124affbf80622f73e634eb" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060d98b7b1844179b502b5ed460817d8"
I20260929 02:55:12.257624 17483 raft_consensus.cc:3037] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:12.257687 17344 raft_consensus.cc:3037] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:12.260015 17483 raft_consensus.cc:2454] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 1.
I20260929 02:55:12.260042 17344 raft_consensus.cc:2454] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 1.
I20260929 02:55:12.260502 17548 leader_election.cc:302] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2674752f524e4805ac74e3cce351a8d9, a62c56ed5cf5477b8c79073ab328546c; no voters:
I20260929 02:55:12.260826 17692 raft_consensus.cc:2781] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:12.261260 17692 raft_consensus.cc:686] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 1 LEADER]: Becoming Leader. State: Replica: a62c56ed5cf5477b8c79073ab328546c, State: Running, Role: LEADER
I20260929 02:55:12.261821 17692 consensus_queue.cc:237] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.266352 17182 catalog_manager.cc:5495] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c reported cstate change: term changed from 0 to 1, leader changed from <none> to a62c56ed5cf5477b8c79073ab328546c (127.15.18.67). New cstate: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } health_report { overall_health: HEALTHY } } }
Sep 29 02:55:12 dist-test-slave-q5f0 krb5kdc[17137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650510, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:12.331219 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.318071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34323 (local address 127.0.0.1:33872)
0929 02:55:12.318604 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:12.318620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:12.318751 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:12.319043 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:12.319046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:12.319753 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:12.320013 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.320023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.320985 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.320988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.322093 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.322107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.323122 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.323127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.323256 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.323905 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:12.323922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:12.326099 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:12.328890 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:12.328901 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:12.328908 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:12.329332 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:12.329680 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:12.329683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:12.329686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:12.329823 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:12.330345 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:12.330351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:12.330539 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 02:55:12.330684 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 02:55:12.330908 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":268,"threads_started":1}
I20260929 02:55:12.331254 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.318025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.15.18.126:34323)
0929 02:55:12.318181 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:12.318185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:12.318703 (+ 518us) server_negotiation.cc:407] Connection header received
0929 02:55:12.318840 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:12.318844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:12.318933 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:12.319015 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:12.320155 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.320877 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.322252 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.322946 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.323058 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.326299 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:12.326305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:12.326308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:12.326338 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:12.328753 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:12.329440 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:12.329444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:12.329446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:12.329588 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:12.329914 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:12.329917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:12.329919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:12.330204 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:12.330310 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:12.330767 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 02:55:12.331026 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":53}
I20260929 02:55:12.336575 17182 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:33872:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:55:12.337631 17182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:12.344645 17319 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8f7429a2774fb19f99fd5b9267e4d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1267b1e2606547db97ef51dbd1c42b27]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:55:12.345019 17597 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8f7429a2774fb19f99fd5b9267e4d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1267b1e2606547db97ef51dbd1c42b27]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:55:12.345747 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8f7429a2774fb19f99fd5b9267e4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:12.344741 17462 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8f7429a2774fb19f99fd5b9267e4d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1267b1e2606547db97ef51dbd1c42b27]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:55:12.346268 17462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8f7429a2774fb19f99fd5b9267e4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:12.346653 17597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8f7429a2774fb19f99fd5b9267e4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:12.352423 17686 tablet_bootstrap.cc:492] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: Bootstrap starting.
I20260929 02:55:12.353138 17688 tablet_bootstrap.cc:492] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: Bootstrap starting.
I20260929 02:55:12.354038 17686 tablet_bootstrap.cc:654] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:12.354758 17688 tablet_bootstrap.cc:654] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:12.355760 17687 tablet_bootstrap.cc:492] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: Bootstrap starting.
I20260929 02:55:12.356469 17686 tablet_bootstrap.cc:492] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: No bootstrap required, opened a new log
I20260929 02:55:12.356638 17686 ts_tablet_manager.cc:1397] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 02:55:12.356784 17688 tablet_bootstrap.cc:492] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: No bootstrap required, opened a new log
I20260929 02:55:12.356927 17688 ts_tablet_manager.cc:1397] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 02:55:12.357337 17687 tablet_bootstrap.cc:654] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:12.357436 17686 raft_consensus.cc:348] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.357563 17688 raft_consensus.cc:348] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.357708 17686 raft_consensus.cc:374] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:12.357793 17688 raft_consensus.cc:374] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:12.357798 17686 raft_consensus.cc:729] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62c56ed5cf5477b8c79073ab328546c, State: Initialized, Role: FOLLOWER
I20260929 02:55:12.357892 17688 raft_consensus.cc:729] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2674752f524e4805ac74e3cce351a8d9, State: Initialized, Role: FOLLOWER
I20260929 02:55:12.358027 17686 consensus_queue.cc:260] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.358144 17688 consensus_queue.cc:260] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.358616 17686 ts_tablet_manager.cc:1428] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:12.358592 17694 tablet_replica.cc:439] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } } }
I20260929 02:55:12.358963 17694 tablet_replica.cc:442] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:12.358919 17692 tablet_replica.cc:439] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } } }
I20260929 02:55:12.359185 17692 tablet_replica.cc:442] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:12.360180 17687 tablet_bootstrap.cc:492] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: No bootstrap required, opened a new log
I20260929 02:55:12.360324 17687 ts_tablet_manager.cc:1397] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:55:12.360399 17688 ts_tablet_manager.cc:1428] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:55:12.361408 17687 raft_consensus.cc:348] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.361662 17687 raft_consensus.cc:374] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:12.361760 17687 raft_consensus.cc:729] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060d98b7b1844179b502b5ed460817d8, State: Initialized, Role: FOLLOWER
I20260929 02:55:12.361985 17687 consensus_queue.cc:260] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.362368 17693 tablet_replica.cc:439] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } } }
I20260929 02:55:12.362653 17693 tablet_replica.cc:442] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:12.363476 17687 ts_tablet_manager.cc:1428] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:55:12.488665 17692 raft_consensus.cc:482] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:12.488888 17692 raft_consensus.cc:504] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.489463 17692 leader_election.cc:288] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685), 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873)
I20260929 02:55:12.490119 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8f7429a2774fb19f99fd5b9267e4d1" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060d98b7b1844179b502b5ed460817d8" is_pre_election: true
I20260929 02:55:12.490119 17483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8f7429a2774fb19f99fd5b9267e4d1" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2674752f524e4805ac74e3cce351a8d9" is_pre_election: true
I20260929 02:55:12.490391 17344 raft_consensus.cc:2454] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 0.
I20260929 02:55:12.490391 17483 raft_consensus.cc:2454] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 0.
I20260929 02:55:12.490818 17548 leader_election.cc:302] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 060d98b7b1844179b502b5ed460817d8, 2674752f524e4805ac74e3cce351a8d9, a62c56ed5cf5477b8c79073ab328546c; no voters:
I20260929 02:55:12.491135 17692 raft_consensus.cc:2781] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:12.491254 17692 raft_consensus.cc:482] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:12.491386 17692 raft_consensus.cc:3037] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:12.493520 17692 raft_consensus.cc:504] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.494165 17692 leader_election.cc:288] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 election: Requested vote from peers 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685), 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873)
I20260929 02:55:12.494745 17483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8f7429a2774fb19f99fd5b9267e4d1" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2674752f524e4805ac74e3cce351a8d9"
I20260929 02:55:12.494825 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8f7429a2774fb19f99fd5b9267e4d1" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060d98b7b1844179b502b5ed460817d8"
I20260929 02:55:12.495000 17483 raft_consensus.cc:3037] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:12.495054 17344 raft_consensus.cc:3037] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:12.497225 17483 raft_consensus.cc:2454] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 1.
I20260929 02:55:12.497704 17548 leader_election.cc:302] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2674752f524e4805ac74e3cce351a8d9, a62c56ed5cf5477b8c79073ab328546c; no voters:
I20260929 02:55:12.497869 17344 raft_consensus.cc:2454] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 1.
I20260929 02:55:12.497964 17692 raft_consensus.cc:2781] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:12.498132 17692 raft_consensus.cc:686] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [term 1 LEADER]: Becoming Leader. State: Replica: a62c56ed5cf5477b8c79073ab328546c, State: Running, Role: LEADER
I20260929 02:55:12.498405 17692 consensus_queue.cc:237] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.500103 17696 tablet_replica.cc:439] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: TxnStatusTablet state changed. Reason: New leader a62c56ed5cf5477b8c79073ab328546c. Latest consensus state: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } } }
I20260929 02:55:12.500371 17696 tablet_replica.cc:442] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:55:12.501471 17181 catalog_manager.cc:5495] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c reported cstate change: term changed from 0 to 1, leader changed from <none> to a62c56ed5cf5477b8c79073ab328546c (127.15.18.67). New cstate: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } health_report { overall_health: HEALTHY } } }
I20260929 02:55:12.511978 17708 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:55:12.512331 17708 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:55:12.522876 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.516651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46137 (local address 127.0.0.1:53974)
0929 02:55:12.516880 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:12.516900 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:12.517053 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:12.517518 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:12.517521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:12.517531 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:12.517690 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.517698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.519136 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.519140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.520202 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.520214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.521215 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.521219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.521385 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.521444 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:12.522439 (+ 995us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:12.522458 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 02:55:12.522532 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:55:12.522650 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 02:55:12.522938 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.516766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.15.18.67:46137)
0929 02:55:12.517099 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:12.517104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:12.517125 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:55:12.517228 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:12.517234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:12.517326 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:12.517459 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:12.517829 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.518996 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.520340 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.521058 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.521177 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.521633 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:12.522324 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:12.522475 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:12.522568 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 02:55:12.522653 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260929 02:55:12.530705 17483 raft_consensus.cc:1261] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Refusing update from remote peer a62c56ed5cf5477b8c79073ab328546c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:12.530705 17344 raft_consensus.cc:1261] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Refusing update from remote peer a62c56ed5cf5477b8c79073ab328546c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:12.531756 17696 consensus_queue.cc:1035] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Connected to new peer: Peer: permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:12.532131 17692 consensus_queue.cc:1035] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:12.540382 17694 tablet_replica.cc:439] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: TxnStatusTablet state changed. Reason: New leader a62c56ed5cf5477b8c79073ab328546c. Latest consensus state: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } } }
I20260929 02:55:12.540726 17694 tablet_replica.cc:442] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:12.544086 17711 mvcc.cc:205] Tried to move back new op lower bound from 7334504499315798016 to 7334504499196231680. Current Snapshot: MvccSnapshot[applied={T|T < 7334504499315798016}]
I20260929 02:55:12.544798 17692 tablet_replica.cc:439] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } } }
I20260929 02:55:12.545136 17692 tablet_replica.cc:442] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:55:12.545866 17712 mvcc.cc:205] Tried to move back new op lower bound from 7334504499315798016 to 7334504499196231680. Current Snapshot: MvccSnapshot[applied={T|T < 7334504499315798016}]
I20260929 02:55:12.549085 17694 tablet_replica.cc:439] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } } }
I20260929 02:55:12.549346 17694 tablet_replica.cc:442] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 2674752f524e4805ac74e3cce351a8d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:12.553543 17693 tablet_replica.cc:439] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: TxnStatusTablet state changed. Reason: New leader a62c56ed5cf5477b8c79073ab328546c. Latest consensus state: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } } }
I20260929 02:55:12.553849 17693 tablet_replica.cc:442] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:12.555408 17696 tablet_replica.cc:439] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } } }
I20260929 02:55:12.555759 17696 tablet_replica.cc:442] T 1c8f7429a2774fb19f99fd5b9267e4d1 P a62c56ed5cf5477b8c79073ab328546c: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:55:12.566118 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.556632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.15.18.67:46137)
0929 02:55:12.556772 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:12.556778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:12.556796 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:12.556864 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:12.556869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:12.556950 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:12.557041 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:12.558748 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.560711 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.563164 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.563905 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.564047 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.564816 (+ 769us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:12.564978 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:12.565060 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:12.565648 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 02:55:12.565745 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 02:55:12.566936 17714 mvcc.cc:205] Tried to move back new op lower bound from 7334504499315798016 to 7334504499196231680. Current Snapshot: MvccSnapshot[applied={T|T < 7334504499315798016}]
I20260929 02:55:12.567728 17693 tablet_replica.cc:439] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } } }
I20260929 02:55:12.567935 17693 tablet_replica.cc:442] T 1c8f7429a2774fb19f99fd5b9267e4d1 P 060d98b7b1844179b502b5ed460817d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:12.584719 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.578202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:52207 (local address 127.15.18.67:46137)
0929 02:55:12.578406 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:12.578411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:12.578938 (+ 527us) server_negotiation.cc:407] Connection header received
0929 02:55:12.579090 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:12.579095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:12.579175 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:12.579270 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:12.579836 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.580949 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.582428 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.583111 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.583253 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.583843 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:12.584021 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:12.584111 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:12.584298 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 02:55:12.584385 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:55:12.584815 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:12.578200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46137 (local address 127.15.18.67:52207)
0929 02:55:12.578835 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:12.578854 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:12.578976 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:12.579312 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:12.579316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:12.579333 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:12.579696 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.579708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.581084 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.581088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.582288 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:12.582301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:12.583357 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:12.583363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:12.583565 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:12.583644 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:12.584167 (+ 523us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:12.584178 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:55:12.584258 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 02:55:12.584382 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":332,"threads_started":1}
I20260929 02:55:12.594441 17482 raft_consensus.cc:1261] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Refusing update from remote peer a62c56ed5cf5477b8c79073ab328546c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:12.594615 17344 raft_consensus.cc:1261] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Refusing update from remote peer a62c56ed5cf5477b8c79073ab328546c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:12.595011 17719 consensus_queue.cc:1035] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:12.595350 17692 consensus_queue.cc:1035] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Connected to new peer: Peer: permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:12.661576 17181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:12.661913 17181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:12.665421 17181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86: Sending DeleteTablet for 3 replicas of tablet 0eeea682dd124affbf80622f73e634eb
I20260929 02:55:12.666532 17319 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eeea682dd124affbf80622f73e634eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:12 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:33126
I20260929 02:55:12.666560 17596 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eeea682dd124affbf80622f73e634eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:12 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:53964
I20260929 02:55:12.666610 17462 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eeea682dd124affbf80622f73e634eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:12 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:40346
I20260929 02:55:12.669070 17181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:12.670241 17181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:12.679795 17728 tablet_replica.cc:330] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: stopping tablet replica
I20260929 02:55:12.680339 17728 raft_consensus.cc:2229] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:12.680943 17461 tablet_service.cc:1467] Processing CreateTablet for tablet 486abd97576240e4b6d4f8e8056ab453 (DEFAULT_TABLE table=test-table [id=243cd692bd4a47ee9440f9edbf3c7d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:12.681062 17728 raft_consensus.cc:2258] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:12.681106 17597 tablet_service.cc:1467] Processing CreateTablet for tablet 486abd97576240e4b6d4f8e8056ab453 (DEFAULT_TABLE table=test-table [id=243cd692bd4a47ee9440f9edbf3c7d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:12.681489 17597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486abd97576240e4b6d4f8e8056ab453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:12.681517 17461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486abd97576240e4b6d4f8e8056ab453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:12.682147 17318 tablet_service.cc:1467] Processing CreateTablet for tablet 486abd97576240e4b6d4f8e8056ab453 (DEFAULT_TABLE table=test-table [id=243cd692bd4a47ee9440f9edbf3c7d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:12.682586 17318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486abd97576240e4b6d4f8e8056ab453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:12.687911 17686 tablet_bootstrap.cc:492] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c: Bootstrap starting.
I20260929 02:55:12.688864 17687 tablet_bootstrap.cc:492] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8: Bootstrap starting.
I20260929 02:55:12.689344 17729 tablet_replica.cc:330] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: stopping tablet replica
I20260929 02:55:12.689642 17686 tablet_bootstrap.cc:654] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:12.689786 17729 raft_consensus.cc:2229] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:12.690135 17729 raft_consensus.cc:2258] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:12.690397 17687 tablet_bootstrap.cc:654] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:12.690696 17688 tablet_bootstrap.cc:492] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9: Bootstrap starting.
I20260929 02:55:12.692320 17688 tablet_bootstrap.cc:654] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:12.693897 17728 ts_tablet_manager.cc:1905] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:12.702721 17687 tablet_bootstrap.cc:492] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8: No bootstrap required, opened a new log
I20260929 02:55:12.702920 17687 ts_tablet_manager.cc:1397] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260929 02:55:12.703752 17687 raft_consensus.cc:348] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.704017 17687 raft_consensus.cc:374] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:12.704102 17687 raft_consensus.cc:729] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060d98b7b1844179b502b5ed460817d8, State: Initialized, Role: FOLLOWER
I20260929 02:55:12.704411 17687 consensus_queue.cc:260] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.704582 17727 tablet_replica.cc:330] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: stopping tablet replica
I20260929 02:55:12.705190 17727 raft_consensus.cc:2229] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:12.707636 17727 raft_consensus.cc:2258] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:12.707911 17687 ts_tablet_manager.cc:1428] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:55:12.709456 17728 ts_tablet_manager.cc:1918] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:12.709708 17728 log.cc:1196] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/0eeea682dd124affbf80622f73e634eb
I20260929 02:55:12.710443 17728 ts_tablet_manager.cc:1939] T 0eeea682dd124affbf80622f73e634eb P a62c56ed5cf5477b8c79073ab328546c: Deleting consensus metadata
I20260929 02:55:12.710721 17727 ts_tablet_manager.cc:1905] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:12.712229 17168 catalog_manager.cc:4841] TS a62c56ed5cf5477b8c79073ab328546c (127.15.18.67:46137): tablet 0eeea682dd124affbf80622f73e634eb (table test-table [id=58b9d5c9ce8644cf81cbc607d9e63929]) successfully deleted
I20260929 02:55:12.714116 17729 ts_tablet_manager.cc:1905] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:12.714732 17688 tablet_bootstrap.cc:492] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9: No bootstrap required, opened a new log
I20260929 02:55:12.714986 17688 ts_tablet_manager.cc:1397] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.008s
I20260929 02:55:12.715919 17688 raft_consensus.cc:348] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.716225 17688 raft_consensus.cc:374] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:12.716784 17688 raft_consensus.cc:729] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2674752f524e4805ac74e3cce351a8d9, State: Initialized, Role: FOLLOWER
I20260929 02:55:12.717303 17686 tablet_bootstrap.cc:492] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c: No bootstrap required, opened a new log
I20260929 02:55:12.717507 17686 ts_tablet_manager.cc:1397] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.007s
I20260929 02:55:12.718168 17727 ts_tablet_manager.cc:1918] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:12.718343 17727 log.cc:1196] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/0eeea682dd124affbf80622f73e634eb
I20260929 02:55:12.718266 17686 raft_consensus.cc:348] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.718551 17686 raft_consensus.cc:374] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:12.718645 17686 raft_consensus.cc:729] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62c56ed5cf5477b8c79073ab328546c, State: Initialized, Role: FOLLOWER
I20260929 02:55:12.718843 17727 ts_tablet_manager.cc:1939] T 0eeea682dd124affbf80622f73e634eb P 060d98b7b1844179b502b5ed460817d8: Deleting consensus metadata
I20260929 02:55:12.718873 17688 consensus_queue.cc:260] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.718917 17686 consensus_queue.cc:260] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.719575 17686 ts_tablet_manager.cc:1428] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260929 02:55:12.721678 17167 catalog_manager.cc:4841] TS 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685): tablet 0eeea682dd124affbf80622f73e634eb (table test-table [id=58b9d5c9ce8644cf81cbc607d9e63929]) successfully deleted
I20260929 02:55:12.722204 17688 ts_tablet_manager.cc:1428] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260929 02:55:12.726994 17729 ts_tablet_manager.cc:1918] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:12.727140 17729 log.cc:1196] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/0eeea682dd124affbf80622f73e634eb
I20260929 02:55:12.727610 17729 ts_tablet_manager.cc:1939] T 0eeea682dd124affbf80622f73e634eb P 2674752f524e4805ac74e3cce351a8d9: Deleting consensus metadata
I20260929 02:55:12.729111 17166 catalog_manager.cc:4841] TS 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873): tablet 0eeea682dd124affbf80622f73e634eb (table test-table [id=58b9d5c9ce8644cf81cbc607d9e63929]) successfully deleted
I20260929 02:55:12.885675 17719 raft_consensus.cc:482] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:12.885917 17719 raft_consensus.cc:504] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.886659 17719 leader_election.cc:288] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685), 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873)
I20260929 02:55:12.887256 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486abd97576240e4b6d4f8e8056ab453" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060d98b7b1844179b502b5ed460817d8" is_pre_election: true
I20260929 02:55:12.887305 17482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486abd97576240e4b6d4f8e8056ab453" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2674752f524e4805ac74e3cce351a8d9" is_pre_election: true
I20260929 02:55:12.887535 17344 raft_consensus.cc:2454] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 0.
I20260929 02:55:12.887562 17482 raft_consensus.cc:2454] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 0.
I20260929 02:55:12.888039 17549 leader_election.cc:302] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 060d98b7b1844179b502b5ed460817d8, 2674752f524e4805ac74e3cce351a8d9, a62c56ed5cf5477b8c79073ab328546c; no voters:
I20260929 02:55:12.888350 17719 raft_consensus.cc:2781] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:12.888460 17719 raft_consensus.cc:482] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:12.888589 17719 raft_consensus.cc:3037] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:12.890837 17719 raft_consensus.cc:504] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.891987 17719 leader_election.cc:288] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 election: Requested vote from peers 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685), 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873)
I20260929 02:55:12.892071 17482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486abd97576240e4b6d4f8e8056ab453" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2674752f524e4805ac74e3cce351a8d9"
I20260929 02:55:12.892298 17482 raft_consensus.cc:3037] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:12.892421 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486abd97576240e4b6d4f8e8056ab453" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060d98b7b1844179b502b5ed460817d8"
I20260929 02:55:12.892609 17344 raft_consensus.cc:3037] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:12.895272 17344 raft_consensus.cc:2454] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 1.
I20260929 02:55:12.895259 17482 raft_consensus.cc:2454] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 1.
I20260929 02:55:12.895742 17549 leader_election.cc:302] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060d98b7b1844179b502b5ed460817d8, a62c56ed5cf5477b8c79073ab328546c; no voters:
I20260929 02:55:12.896034 17719 raft_consensus.cc:2781] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:12.896205 17719 raft_consensus.cc:686] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 1 LEADER]: Becoming Leader. State: Replica: a62c56ed5cf5477b8c79073ab328546c, State: Running, Role: LEADER
I20260929 02:55:12.896464 17719 consensus_queue.cc:237] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:12.899291 17180 catalog_manager.cc:5495] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c reported cstate change: term changed from 0 to 1, leader changed from <none> to a62c56ed5cf5477b8c79073ab328546c (127.15.18.67). New cstate: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } health_report { overall_health: HEALTHY } } }
I20260929 02:55:12.960289 17344 raft_consensus.cc:1261] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Refusing update from remote peer a62c56ed5cf5477b8c79073ab328546c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:12.960750 17696 consensus_queue.cc:1035] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Connected to new peer: Peer: permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:12.960898 17483 raft_consensus.cc:1261] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Refusing update from remote peer a62c56ed5cf5477b8c79073ab328546c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:12.961474 17696 consensus_queue.cc:1035] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:55:12.990859 17738 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 02:55:14.592881 17180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:14.593137 17180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:14.597016 17180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86: Sending DeleteTablet for 3 replicas of tablet 486abd97576240e4b6d4f8e8056ab453
I20260929 02:55:14.597795 17462 tablet_service.cc:1514] Processing DeleteTablet for tablet 486abd97576240e4b6d4f8e8056ab453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:14 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:40346
I20260929 02:55:14.597808 17319 tablet_service.cc:1514] Processing DeleteTablet for tablet 486abd97576240e4b6d4f8e8056ab453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:14 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:33126
I20260929 02:55:14.597884 17596 tablet_service.cc:1514] Processing DeleteTablet for tablet 486abd97576240e4b6d4f8e8056ab453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:14 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:53964
I20260929 02:55:14.610262 17760 tablet_replica.cc:330] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9: stopping tablet replica
I20260929 02:55:14.610626 17760 raft_consensus.cc:2229] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:14.610971 17760 raft_consensus.cc:2258] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:14.612126 17760 ts_tablet_manager.cc:1905] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:14.613112 17759 tablet_replica.cc:330] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8: stopping tablet replica
I20260929 02:55:14.613405 17759 raft_consensus.cc:2229] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:14.613694 17759 raft_consensus.cc:2258] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:14.614266 17761 tablet_replica.cc:330] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c: stopping tablet replica
I20260929 02:55:14.614486 17761 raft_consensus.cc:2229] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:14.614861 17761 raft_consensus.cc:2258] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:14.615279 17759 ts_tablet_manager.cc:1905] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:14.615875 17761 ts_tablet_manager.cc:1905] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:14.618891 17760 ts_tablet_manager.cc:1918] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:55:14.619117 17760 log.cc:1196] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/486abd97576240e4b6d4f8e8056ab453
I20260929 02:55:14.619642 17760 ts_tablet_manager.cc:1939] T 486abd97576240e4b6d4f8e8056ab453 P 2674752f524e4805ac74e3cce351a8d9: Deleting consensus metadata
I20260929 02:55:14.620865 17166 catalog_manager.cc:4841] TS 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873): tablet 486abd97576240e4b6d4f8e8056ab453 (table test-table [id=243cd692bd4a47ee9440f9edbf3c7d15]) successfully deleted
I20260929 02:55:14.621872 17759 ts_tablet_manager.cc:1918] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:55:14.622012 17759 log.cc:1196] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/486abd97576240e4b6d4f8e8056ab453
I20260929 02:55:14.621989 17761 ts_tablet_manager.cc:1918] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:55:14.622107 17761 log.cc:1196] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/486abd97576240e4b6d4f8e8056ab453
I20260929 02:55:14.622464 17759 ts_tablet_manager.cc:1939] T 486abd97576240e4b6d4f8e8056ab453 P 060d98b7b1844179b502b5ed460817d8: Deleting consensus metadata
I20260929 02:55:14.622495 17761 ts_tablet_manager.cc:1939] T 486abd97576240e4b6d4f8e8056ab453 P a62c56ed5cf5477b8c79073ab328546c: Deleting consensus metadata
I20260929 02:55:14.623488 17168 catalog_manager.cc:4841] TS a62c56ed5cf5477b8c79073ab328546c (127.15.18.67:46137): tablet 486abd97576240e4b6d4f8e8056ab453 (table test-table [id=243cd692bd4a47ee9440f9edbf3c7d15]) successfully deleted
I20260929 02:55:14.623502 17167 catalog_manager.cc:4841] TS 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685): tablet 486abd97576240e4b6d4f8e8056ab453 (table test-table [id=243cd692bd4a47ee9440f9edbf3c7d15]) successfully deleted
Sep 29 02:55:14 dist-test-slave-q5f0 krb5kdc[17137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650514, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:55:14 dist-test-slave-q5f0 krb5kdc[17137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650514, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:14.656827 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.640653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.15.18.126:34323)
0929 02:55:14.641148 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:14.641153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:14.641176 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:55:14.641344 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:14.641348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:14.641408 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:14.641508 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:14.642855 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.643678 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.644792 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.645177 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.645275 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.652278 (+ 7003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:14.652283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:14.652287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:14.652322 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:14.654736 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:14.655348 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:14.655352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:14.655354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:14.655474 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:14.655783 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:14.655786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:14.655788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:14.656049 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:14.656217 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:14.656416 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 02:55:14.656577 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":262,"threads_started":1}
I20260929 02:55:14.660389 17180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:14.660965 17180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:14.666949 17319 tablet_service.cc:1467] Processing CreateTablet for tablet dded7416a2a6450589758b41b99c9eb0 (DEFAULT_TABLE table=test-table [id=d4989cb62b74470c8590cba48590281e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:14.666949 17596 tablet_service.cc:1467] Processing CreateTablet for tablet dded7416a2a6450589758b41b99c9eb0 (DEFAULT_TABLE table=test-table [id=d4989cb62b74470c8590cba48590281e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:14.667037 17462 tablet_service.cc:1467] Processing CreateTablet for tablet dded7416a2a6450589758b41b99c9eb0 (DEFAULT_TABLE table=test-table [id=d4989cb62b74470c8590cba48590281e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:14.667376 17596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dded7416a2a6450589758b41b99c9eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:14.667376 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dded7416a2a6450589758b41b99c9eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:14.667452 17462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dded7416a2a6450589758b41b99c9eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:14.673246 17770 tablet_bootstrap.cc:492] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9: Bootstrap starting.
I20260929 02:55:14.673894 17771 tablet_bootstrap.cc:492] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c: Bootstrap starting.
I20260929 02:55:14.674623 17772 tablet_bootstrap.cc:492] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8: Bootstrap starting.
I20260929 02:55:14.675768 17771 tablet_bootstrap.cc:654] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:14.675810 17770 tablet_bootstrap.cc:654] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:14.676348 17772 tablet_bootstrap.cc:654] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:14.677902 17771 tablet_bootstrap.cc:492] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c: No bootstrap required, opened a new log
I20260929 02:55:14.678063 17771 ts_tablet_manager.cc:1397] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 02:55:14.678787 17772 tablet_bootstrap.cc:492] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8: No bootstrap required, opened a new log
I20260929 02:55:14.678828 17770 tablet_bootstrap.cc:492] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9: No bootstrap required, opened a new log
I20260929 02:55:14.678941 17772 ts_tablet_manager.cc:1397] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:55:14.678951 17770 ts_tablet_manager.cc:1397] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 02:55:14.679045 17771 raft_consensus.cc:348] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.679311 17771 raft_consensus.cc:374] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:14.679425 17771 raft_consensus.cc:729] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62c56ed5cf5477b8c79073ab328546c, State: Initialized, Role: FOLLOWER
I20260929 02:55:14.679656 17770 raft_consensus.cc:348] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.679913 17770 raft_consensus.cc:374] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:14.680027 17770 raft_consensus.cc:729] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2674752f524e4805ac74e3cce351a8d9, State: Initialized, Role: FOLLOWER
I20260929 02:55:14.679905 17772 raft_consensus.cc:348] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.679875 17771 consensus_queue.cc:260] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.680198 17772 raft_consensus.cc:374] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:14.680326 17772 raft_consensus.cc:729] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060d98b7b1844179b502b5ed460817d8, State: Initialized, Role: FOLLOWER
I20260929 02:55:14.680296 17770 consensus_queue.cc:260] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.680608 17772 consensus_queue.cc:260] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.680841 17771 ts_tablet_manager.cc:1428] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:55:14.684072 17770 ts_tablet_manager.cc:1428] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260929 02:55:14.684818 17772 ts_tablet_manager.cc:1428] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 02:55:14.725589 17739 raft_consensus.cc:482] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:14.725777 17739 raft_consensus.cc:504] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.726497 17739 leader_election.cc:288] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685), 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873)
I20260929 02:55:14.727013 17483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dded7416a2a6450589758b41b99c9eb0" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2674752f524e4805ac74e3cce351a8d9" is_pre_election: true
I20260929 02:55:14.727013 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dded7416a2a6450589758b41b99c9eb0" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060d98b7b1844179b502b5ed460817d8" is_pre_election: true
I20260929 02:55:14.727252 17343 raft_consensus.cc:2454] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 0.
I20260929 02:55:14.727262 17483 raft_consensus.cc:2454] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 0.
I20260929 02:55:14.727710 17548 leader_election.cc:302] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 060d98b7b1844179b502b5ed460817d8, 2674752f524e4805ac74e3cce351a8d9, a62c56ed5cf5477b8c79073ab328546c; no voters:
I20260929 02:55:14.727988 17739 raft_consensus.cc:2781] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:14.728092 17739 raft_consensus.cc:482] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:14.728178 17739 raft_consensus.cc:3037] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:14.730176 17739 raft_consensus.cc:504] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.730634 17739 leader_election.cc:288] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 election: Requested vote from peers 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685), 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873)
I20260929 02:55:14.731070 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dded7416a2a6450589758b41b99c9eb0" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060d98b7b1844179b502b5ed460817d8"
I20260929 02:55:14.731097 17483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dded7416a2a6450589758b41b99c9eb0" candidate_uuid: "a62c56ed5cf5477b8c79073ab328546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2674752f524e4805ac74e3cce351a8d9"
I20260929 02:55:14.731257 17343 raft_consensus.cc:3037] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:14.731257 17483 raft_consensus.cc:3037] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:14.733162 17343 raft_consensus.cc:2454] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 1.
I20260929 02:55:14.733191 17483 raft_consensus.cc:2454] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62c56ed5cf5477b8c79073ab328546c in term 1.
I20260929 02:55:14.733508 17549 leader_election.cc:302] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060d98b7b1844179b502b5ed460817d8, a62c56ed5cf5477b8c79073ab328546c; no voters:
I20260929 02:55:14.733762 17739 raft_consensus.cc:2781] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:14.733901 17739 raft_consensus.cc:686] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 1 LEADER]: Becoming Leader. State: Replica: a62c56ed5cf5477b8c79073ab328546c, State: Running, Role: LEADER
I20260929 02:55:14.734174 17739 consensus_queue.cc:237] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.736632 17180 catalog_manager.cc:5495] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c reported cstate change: term changed from 0 to 1, leader changed from <none> to a62c56ed5cf5477b8c79073ab328546c (127.15.18.67). New cstate: current_term: 1 leader_uuid: "a62c56ed5cf5477b8c79073ab328546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } health_report { overall_health: HEALTHY } } }
I20260929 02:55:14.771044 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.766239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.15.18.67:46137)
0929 02:55:14.766800 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:14.766805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:14.766823 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:14.766880 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:14.766883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:14.766947 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:14.767024 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:14.767390 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.768174 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.769421 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.769832 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.769971 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.770348 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:14.770457 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:14.770528 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:14.770664 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 02:55:14.770738 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":303,"threads_started":1}
I20260929 02:55:14.779116 17483 raft_consensus.cc:1261] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Refusing update from remote peer a62c56ed5cf5477b8c79073ab328546c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:14.779140 17343 raft_consensus.cc:1261] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Refusing update from remote peer a62c56ed5cf5477b8c79073ab328546c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:14.779596 17739 consensus_queue.cc:1035] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Connected to new peer: Peer: permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:14.779904 17742 consensus_queue.cc:1035] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:14.818413 17181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:14.818616 17181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:14.821214 17181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86: Sending DeleteTablet for 3 replicas of tablet dded7416a2a6450589758b41b99c9eb0
I20260929 02:55:14.821916 17462 tablet_service.cc:1514] Processing DeleteTablet for tablet dded7416a2a6450589758b41b99c9eb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:14 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:40346
I20260929 02:55:14.822009 17319 tablet_service.cc:1514] Processing DeleteTablet for tablet dded7416a2a6450589758b41b99c9eb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:14 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:33126
I20260929 02:55:14.822125 17597 tablet_service.cc:1514] Processing DeleteTablet for tablet dded7416a2a6450589758b41b99c9eb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:14 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:53964
I20260929 02:55:14.822235 17760 tablet_replica.cc:330] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9: stopping tablet replica
I20260929 02:55:14.822297 17759 tablet_replica.cc:330] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8: stopping tablet replica
I20260929 02:55:14.822359 17761 tablet_replica.cc:330] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c: stopping tablet replica
I20260929 02:55:14.822413 17760 raft_consensus.cc:2229] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:14.822499 17759 raft_consensus.cc:2229] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:14.822518 17761 raft_consensus.cc:2229] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:14.822575 17760 raft_consensus.cc:2258] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:14.822666 17759 raft_consensus.cc:2258] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:14.822795 17761 raft_consensus.cc:2258] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:14.823639 17759 ts_tablet_manager.cc:1905] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:14.823793 17760 ts_tablet_manager.cc:1905] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:14.823856 17761 ts_tablet_manager.cc:1905] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:14.823642 17181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:14.824292 17181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:14.830084 17760 ts_tablet_manager.cc:1918] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:14.830330 17760 log.cc:1196] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/dded7416a2a6450589758b41b99c9eb0
I20260929 02:55:14.830782 17760 ts_tablet_manager.cc:1939] T dded7416a2a6450589758b41b99c9eb0 P 2674752f524e4805ac74e3cce351a8d9: Deleting consensus metadata
I20260929 02:55:14.830860 17759 ts_tablet_manager.cc:1918] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:14.831018 17759 log.cc:1196] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/dded7416a2a6450589758b41b99c9eb0
I20260929 02:55:14.831601 17597 tablet_service.cc:1467] Processing CreateTablet for tablet b2f0e1503ef74ca5bc3e9eb18629e75f (DEFAULT_TABLE table=test-table [id=c5be1a7810b44a67b76537ad21fa3862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:14.831853 17759 ts_tablet_manager.cc:1939] T dded7416a2a6450589758b41b99c9eb0 P 060d98b7b1844179b502b5ed460817d8: Deleting consensus metadata
I20260929 02:55:14.831785 17462 tablet_service.cc:1467] Processing CreateTablet for tablet b2f0e1503ef74ca5bc3e9eb18629e75f (DEFAULT_TABLE table=test-table [id=c5be1a7810b44a67b76537ad21fa3862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:14.832079 17597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f0e1503ef74ca5bc3e9eb18629e75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:14.832161 17462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f0e1503ef74ca5bc3e9eb18629e75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:14.834630 17166 catalog_manager.cc:4841] TS 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873): tablet dded7416a2a6450589758b41b99c9eb0 (table test-table [id=d4989cb62b74470c8590cba48590281e]) successfully deleted
I20260929 02:55:14.835000 17761 ts_tablet_manager.cc:1918] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:14.835193 17761 log.cc:1196] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/dded7416a2a6450589758b41b99c9eb0
I20260929 02:55:14.835650 17761 ts_tablet_manager.cc:1939] T dded7416a2a6450589758b41b99c9eb0 P a62c56ed5cf5477b8c79073ab328546c: Deleting consensus metadata
I20260929 02:55:14.835626 17167 catalog_manager.cc:4841] TS 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685): tablet dded7416a2a6450589758b41b99c9eb0 (table test-table [id=d4989cb62b74470c8590cba48590281e]) successfully deleted
I20260929 02:55:14.835893 17319 tablet_service.cc:1467] Processing CreateTablet for tablet b2f0e1503ef74ca5bc3e9eb18629e75f (DEFAULT_TABLE table=test-table [id=c5be1a7810b44a67b76537ad21fa3862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:14.836441 17319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f0e1503ef74ca5bc3e9eb18629e75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:14.837010 17168 catalog_manager.cc:4841] TS a62c56ed5cf5477b8c79073ab328546c (127.15.18.67:46137): tablet dded7416a2a6450589758b41b99c9eb0 (table test-table [id=d4989cb62b74470c8590cba48590281e]) successfully deleted
I20260929 02:55:14.838627 17771 tablet_bootstrap.cc:492] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c: Bootstrap starting.
I20260929 02:55:14.840281 17771 tablet_bootstrap.cc:654] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:14.840444 17770 tablet_bootstrap.cc:492] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9: Bootstrap starting.
I20260929 02:55:14.841904 17770 tablet_bootstrap.cc:654] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:14.842285 17772 tablet_bootstrap.cc:492] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8: Bootstrap starting.
I20260929 02:55:14.842370 17771 tablet_bootstrap.cc:492] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c: No bootstrap required, opened a new log
I20260929 02:55:14.842482 17771 ts_tablet_manager.cc:1397] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 02:55:14.843199 17771 raft_consensus.cc:348] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.843432 17771 raft_consensus.cc:374] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:14.843528 17771 raft_consensus.cc:729] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62c56ed5cf5477b8c79073ab328546c, State: Initialized, Role: FOLLOWER
I20260929 02:55:14.843660 17772 tablet_bootstrap.cc:654] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:14.843756 17771 consensus_queue.cc:260] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.843917 17770 tablet_bootstrap.cc:492] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9: No bootstrap required, opened a new log
I20260929 02:55:14.844054 17770 ts_tablet_manager.cc:1397] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 02:55:14.844224 17771 ts_tablet_manager.cc:1428] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:14.844734 17770 raft_consensus.cc:348] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.844964 17770 raft_consensus.cc:374] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:14.845045 17770 raft_consensus.cc:729] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2674752f524e4805ac74e3cce351a8d9, State: Initialized, Role: FOLLOWER
I20260929 02:55:14.845275 17770 consensus_queue.cc:260] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.846243 17770 ts_tablet_manager.cc:1428] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:14.847632 17772 tablet_bootstrap.cc:492] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8: No bootstrap required, opened a new log
I20260929 02:55:14.847795 17772 ts_tablet_manager.cc:1397] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 02:55:14.848464 17772 raft_consensus.cc:348] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.848696 17772 raft_consensus.cc:374] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:14.848780 17772 raft_consensus.cc:729] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060d98b7b1844179b502b5ed460817d8, State: Initialized, Role: FOLLOWER
I20260929 02:55:14.848994 17772 consensus_queue.cc:260] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.849423 17772 ts_tablet_manager.cc:1428] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:55:14.880707 17777 raft_consensus.cc:482] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:14.880966 17777 raft_consensus.cc:504] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.882624 17777 leader_election.cc:288] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873), a62c56ed5cf5477b8c79073ab328546c (127.15.18.67:46137)
I20260929 02:55:14.889101 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.882965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:57629 (local address 127.15.18.66:44873)
0929 02:55:14.884010 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:14.884014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:14.884032 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:14.884085 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:14.884089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:14.884142 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:14.884220 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:14.884779 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.885547 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.887246 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.888083 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.888337 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.888410 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:14.888648 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 02:55:14.888776 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":785,"threads_started":1}
I20260929 02:55:14.889168 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.882932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:44873 (local address 127.15.18.65:57629)
0929 02:55:14.883713 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:14.883730 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:14.883861 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:14.884262 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:14.884267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:14.884293 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:14.884605 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:14.884614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.885670 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.885674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:14.887102 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:14.887113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.888333 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.888340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:14.888504 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.888528 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:55:14.888614 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 02:55:14.888746 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"spinlock_wait_cycles":1920,"thread_start_us":447,"threads_started":1}
I20260929 02:55:14.889195 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.883045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:52515 (local address 127.15.18.67:46137)
0929 02:55:14.883238 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:14.883245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:14.884369 (+ 1124us) server_negotiation.cc:407] Connection header received
0929 02:55:14.884572 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:14.884576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:14.884632 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:14.884707 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:14.885202 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.885933 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.887562 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.888340 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.888480 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.888554 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:14.888831 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 02:55:14.888954 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:55:14.889305 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.882934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46137 (local address 127.15.18.65:52515)
0929 02:55:14.884264 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:14.884277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:14.884433 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:14.884759 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:14.884763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:14.884784 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:14.885061 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:14.885071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.886057 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.886062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:14.887421 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:14.887432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.888557 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.888563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:14.888697 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.888719 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:55:14.888796 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:55:14.888911 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"mutex_wait_us":73,"spinlock_wait_cycles":1280,"thread_start_us":1002,"threads_started":1}
I20260929 02:55:14.890025 17483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f0e1503ef74ca5bc3e9eb18629e75f" candidate_uuid: "060d98b7b1844179b502b5ed460817d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2674752f524e4805ac74e3cce351a8d9" is_pre_election: true
I20260929 02:55:14.890098 17617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f0e1503ef74ca5bc3e9eb18629e75f" candidate_uuid: "060d98b7b1844179b502b5ed460817d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62c56ed5cf5477b8c79073ab328546c" is_pre_election: true
I20260929 02:55:14.890268 17483 raft_consensus.cc:2454] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060d98b7b1844179b502b5ed460817d8 in term 0.
I20260929 02:55:14.890487 17617 raft_consensus.cc:2454] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060d98b7b1844179b502b5ed460817d8 in term 0.
I20260929 02:55:14.890722 17269 leader_election.cc:302] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060d98b7b1844179b502b5ed460817d8, 2674752f524e4805ac74e3cce351a8d9; no voters:
I20260929 02:55:14.891126 17777 raft_consensus.cc:2781] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:14.891263 17777 raft_consensus.cc:482] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:14.891357 17777 raft_consensus.cc:3037] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:14.893477 17777 raft_consensus.cc:504] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.894027 17777 leader_election.cc:288] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [CANDIDATE]: Term 1 election: Requested vote from peers 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873), a62c56ed5cf5477b8c79073ab328546c (127.15.18.67:46137)
I20260929 02:55:14.894480 17483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f0e1503ef74ca5bc3e9eb18629e75f" candidate_uuid: "060d98b7b1844179b502b5ed460817d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2674752f524e4805ac74e3cce351a8d9"
I20260929 02:55:14.894518 17617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f0e1503ef74ca5bc3e9eb18629e75f" candidate_uuid: "060d98b7b1844179b502b5ed460817d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62c56ed5cf5477b8c79073ab328546c"
I20260929 02:55:14.894660 17483 raft_consensus.cc:3037] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:14.894677 17617 raft_consensus.cc:3037] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:14.896587 17483 raft_consensus.cc:2454] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060d98b7b1844179b502b5ed460817d8 in term 1.
I20260929 02:55:14.896694 17617 raft_consensus.cc:2454] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060d98b7b1844179b502b5ed460817d8 in term 1.
I20260929 02:55:14.896967 17269 leader_election.cc:302] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060d98b7b1844179b502b5ed460817d8, 2674752f524e4805ac74e3cce351a8d9; no voters:
I20260929 02:55:14.897248 17777 raft_consensus.cc:2781] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:14.897635 17777 raft_consensus.cc:686] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 1 LEADER]: Becoming Leader. State: Replica: 060d98b7b1844179b502b5ed460817d8, State: Running, Role: LEADER
I20260929 02:55:14.898002 17777 consensus_queue.cc:237] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } }
I20260929 02:55:14.901129 17181 catalog_manager.cc:5495] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 060d98b7b1844179b502b5ed460817d8 (127.15.18.65). New cstate: current_term: 1 leader_uuid: "060d98b7b1844179b502b5ed460817d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060d98b7b1844179b502b5ed460817d8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 } health_report { overall_health: UNKNOWN } } }
I20260929 02:55:14.921226 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.915793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55562 (local address 127.15.18.65:46685)
0929 02:55:14.916445 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:14.916449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:14.916468 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:55:14.916541 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:14.916544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:14.916616 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:14.916746 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:14.917102 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.917836 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.919124 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.919577 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.919678 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.920064 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:14.920664 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:14.920768 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:14.920807 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 02:55:14.920882 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":400,"threads_started":1}
I20260929 02:55:14.935711 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.930484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:38897 (local address 127.15.18.67:46137)
0929 02:55:14.930633 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:14.930638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:14.931170 (+ 532us) server_negotiation.cc:407] Connection header received
0929 02:55:14.931353 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:14.931357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:14.931420 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:14.931489 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:14.931995 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.932715 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.934032 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.934480 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.934590 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.934991 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:14.935098 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:14.935168 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:14.935304 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 02:55:14.935389 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:55:14.935801 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.930466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46137 (local address 127.15.18.65:38897)
0929 02:55:14.931077 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:14.931091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:14.931224 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:14.931540 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:14.931544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:14.931556 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:14.931862 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:14.931871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.932845 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.932849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:14.933906 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:14.933918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.934632 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.934638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:14.934771 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.934824 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:14.935182 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:14.935193 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:55:14.935260 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 02:55:14.935393 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":308,"threads_started":1}
I20260929 02:55:14.943440 17483 raft_consensus.cc:1261] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Refusing update from remote peer 060d98b7b1844179b502b5ed460817d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:14.943681 17617 raft_consensus.cc:1261] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Refusing update from remote peer 060d98b7b1844179b502b5ed460817d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:14.944094 17791 consensus_queue.cc:1035] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2674752f524e4805ac74e3cce351a8d9" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:14.944479 17777 consensus_queue.cc:1035] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62c56ed5cf5477b8c79073ab328546c" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:14.969717 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.964523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:44001 (local address 127.15.18.65:46685)
0929 02:55:14.964790 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:14.964797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:14.965019 (+ 222us) server_negotiation.cc:407] Connection header received
0929 02:55:14.965406 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:14.965410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:14.965468 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:14.965539 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:14.965994 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.966723 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.968268 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.968678 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.968768 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.969145 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:14.969247 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:14.969319 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:14.969419 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 02:55:14.969495 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260929 02:55:14.969775 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:14.964407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:46685 (local address 127.15.18.67:44001)
0929 02:55:14.964936 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:14.964955 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:14.965086 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:14.965572 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:14.965577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:14.965589 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:14.965860 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:14.965869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.966830 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.966834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:14.968131 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:14.968144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:14.968822 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:14.968826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:14.968944 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:14.968979 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:14.969324 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:14.969331 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:55:14.969392 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 02:55:14.969500 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":255,"threads_started":1}
W20260929 02:55:14.970826 17744 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 02:55:16.575042 17180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:16.575277 17180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:16.579648 17180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01b90c2a475a4e6298f6cc173e856d86: Sending DeleteTablet for 3 replicas of tablet b2f0e1503ef74ca5bc3e9eb18629e75f
I20260929 02:55:16.580502 17462 tablet_service.cc:1514] Processing DeleteTablet for tablet b2f0e1503ef74ca5bc3e9eb18629e75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:16 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:40346
I20260929 02:55:16.580590 17597 tablet_service.cc:1514] Processing DeleteTablet for tablet b2f0e1503ef74ca5bc3e9eb18629e75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:16 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:53964
I20260929 02:55:16.580653 17319 tablet_service.cc:1514] Processing DeleteTablet for tablet b2f0e1503ef74ca5bc3e9eb18629e75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:16 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:33126
I20260929 02:55:16.581326 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 17253
I20260929 02:55:16.595795 17824 tablet_replica.cc:330] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9: stopping tablet replica
I20260929 02:55:16.596150 17824 raft_consensus.cc:2229] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 29 02:55:16 dist-test-slave-q5f0 krb5kdc[17137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650514, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:16.596539 17824 raft_consensus.cc:2258] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:16.597329 17825 tablet_replica.cc:330] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c: stopping tablet replica
I20260929 02:55:16.597599 17825 raft_consensus.cc:2229] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:16.597846 17824 ts_tablet_manager.cc:1905] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:16.597901 17825 raft_consensus.cc:2258] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:16.600268 17825 ts_tablet_manager.cc:1905] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:16.603964 17826 tablet_replica.cc:330] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8: stopping tablet replica
I20260929 02:55:16.604223 17826 raft_consensus.cc:2229] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:16.604700 17826 raft_consensus.cc:2258] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:16.605604 17824 ts_tablet_manager.cc:1918] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:55:16.605777 17826 ts_tablet_manager.cc:1905] T b2f0e1503ef74ca5bc3e9eb18629e75f P 060d98b7b1844179b502b5ed460817d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:16.605947 17824 log.cc:1196] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/b2f0e1503ef74ca5bc3e9eb18629e75f
I20260929 02:55:16.606581 17824 ts_tablet_manager.cc:1939] T b2f0e1503ef74ca5bc3e9eb18629e75f P 2674752f524e4805ac74e3cce351a8d9: Deleting consensus metadata
I20260929 02:55:16.607873 17166 catalog_manager.cc:4841] TS 2674752f524e4805ac74e3cce351a8d9 (127.15.18.66:44873): tablet b2f0e1503ef74ca5bc3e9eb18629e75f (table test-table [id=c5be1a7810b44a67b76537ad21fa3862]) successfully deleted
I20260929 02:55:16.607679 17825 ts_tablet_manager.cc:1918] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:55:16.608268 17825 log.cc:1196] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/b2f0e1503ef74ca5bc3e9eb18629e75f
I20260929 02:55:16.608355 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:16.582738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.15.18.65:46685)
0929 02:55:16.583560 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:16.583567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:16.584288 (+ 721us) server_negotiation.cc:407] Connection header received
0929 02:55:16.584363 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:16.584368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:16.584484 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:16.584576 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:16.586518 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:16.588049 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:16.590392 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:16.591192 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:16.591327 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:16.599566 (+ 8239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:16.599592 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:16.599613 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:16.599672 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:16.603879 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:16.604917 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:16.604926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:16.604930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:16.605116 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:16.605614 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:16.605618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:16.605621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:16.606142 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:16.606437 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:16.607681 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0929 02:55:16.607945 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":544,"threads_started":1}
I20260929 02:55:16.608776 17825 ts_tablet_manager.cc:1939] T b2f0e1503ef74ca5bc3e9eb18629e75f P a62c56ed5cf5477b8c79073ab328546c: Deleting consensus metadata
W20260929 02:55:16.609112 17384 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55566
W20260929 02:55:16.609820 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 17253: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:55:16.609967 17168 catalog_manager.cc:4841] TS a62c56ed5cf5477b8c79073ab328546c (127.15.18.67:46137): tablet b2f0e1503ef74ca5bc3e9eb18629e75f (table test-table [id=c5be1a7810b44a67b76537ad21fa3862]) successfully deleted
I20260929 02:55:16.610038 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 17253
W20260929 02:55:16.627038 17167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.18.65:46685 (error 108)
I20260929 02:55:16.628026 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 17393
I20260929 02:55:16.629354 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:16.628486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:46685 (local address 127.0.0.1:55578)
0929 02:55:16.628984 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:16.629126 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:46685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
W20260929 02:55:16.629586 17167 catalog_manager.cc:4584] TS 060d98b7b1844179b502b5ed460817d8 (127.15.18.65:46685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2f0e1503ef74ca5bc3e9eb18629e75f: Network error: Client connection negotiation failed: client connection to 127.15.18.65:46685: connect: Connection refused (error 111)
Sep 29 02:55:16 dist-test-slave-q5f0 krb5kdc[17137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650514, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:55:16.643229 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:16.628788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.15.18.66:44873)
0929 02:55:16.629271 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:16.629277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:16.629296 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:55:16.629360 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:16.629365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:16.629458 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:16.629597 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:16.630846 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:16.631647 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:16.632887 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:16.633307 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:16.633424 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:16.637652 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:16.637666 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:16.637682 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:16.637740 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:16.640626 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:16.641300 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:16.641313 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:16.641317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:16.641450 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:16.641833 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:16.641837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:16.641839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:16.642297 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:16.642552 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:16.642786 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 02:55:16.642959 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
W20260929 02:55:16.643918 17523 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40352
W20260929 02:55:16.644536 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 17393: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:55:16.644665 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 17393
I20260929 02:55:16.658547 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 17532
Sep 29 02:55:16 dist-test-slave-q5f0 krb5kdc[17137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650514, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:55:16.674192 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:16.659362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.15.18.67:46137)
0929 02:55:16.659839 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:16.659844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:16.659861 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:55:16.659913 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:16.659916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:16.659983 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:16.660081 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:16.661354 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:16.662090 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:16.663356 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:16.663957 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:16.664074 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:16.668475 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:16.668492 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:16.668509 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:16.668551 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:16.671338 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:16.672240 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:16.672249 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:16.672251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:16.672399 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:16.672803 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:16.672806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:16.672808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:16.673187 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:16.673413 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:16.673694 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 02:55:16.673908 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":254,"threads_started":1}
W20260929 02:55:16.674907 17662 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50848
W20260929 02:55:16.675643 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 17532: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:55:16.675818 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 17532
I20260929 02:55:16.683648 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:16.683095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:46685 (local address 127.0.0.1:55594)
0929 02:55:16.683333 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:16.683430 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:46685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 02:55:16.684800 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:16.684384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:46685 (local address 127.0.0.1:55602)
0929 02:55:16.684564 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:16.684654 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:46685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 02:55:16.695940 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 17150
I20260929 02:55:16.697113 17222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:16.812793 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:16.812383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:46685 (local address 127.0.0.1:55606)
0929 02:55:16.812564 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:16.812646 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:46685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 02:55:16.812875 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 17150
2026-09-29T02:55:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8037 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:16 dist-test-slave-q5f0 krb5kdc[17836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:16 dist-test-slave-q5f0 krb5kdc[17836](info): set up 2 sockets
Sep 29 02:55:16 dist-test-slave-q5f0 krb5kdc[17836](info): commencing operation
krb5kdc: starting...
W20260929 02:55:17.815863 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:55:17 dist-test-slave-q5f0 krb5kdc[17836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:55:17.994797 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46691
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:18.305563 17849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:18.305867 17849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:18.305997 17849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:18.313140 17849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:18.313249 17849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:18.313299 17849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:18.313375 17849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:18.313432 17849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:18.323163 17849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:46691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:18.324738 17849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:18.326115 17849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:18.337771 17854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:18.337798 17857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:18.337855 17855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:18.337884 17849 server_base.cc:1034] running on GCE node
I20260929 02:55:18.338979 17849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:18.340381 17849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:18.341599 17849 hybrid_clock.cc:648] HybridClock initialized: now 1790650518341553 us; error 35 us; skew 500 ppm
I20260929 02:55:18.341950 17849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:18 dist-test-slave-q5f0 krb5kdc[17836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:18.344960 17849 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:55:18.346391 17849 webserver.cc:460] Webserver started at http://127.15.18.126:37523/ using document root <none> and password file <none>
I20260929 02:55:18.347054 17849 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:18.347163 17849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:18.347442 17849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:18.350423 17849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "626c55995b06447b9c7120b13eefd738"
format_stamp: "Formatted at 2026-09-29 02:55:18 on dist-test-slave-q5f0"
server_key: "e104820db02b8964cbb60dfebbda1231"
server_key_iv: "be86e90396cd198f30dd9e5e67fde591"
server_key_version: "encryptionkey@0"
I20260929 02:55:18.351219 17849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "626c55995b06447b9c7120b13eefd738"
format_stamp: "Formatted at 2026-09-29 02:55:18 on dist-test-slave-q5f0"
server_key: "e104820db02b8964cbb60dfebbda1231"
server_key_iv: "be86e90396cd198f30dd9e5e67fde591"
server_key_version: "encryptionkey@0"
I20260929 02:55:18.356503 17849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:55:18.360056 17864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:18.361454 17849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:55:18.361620 17849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "626c55995b06447b9c7120b13eefd738"
format_stamp: "Formatted at 2026-09-29 02:55:18 on dist-test-slave-q5f0"
server_key: "e104820db02b8964cbb60dfebbda1231"
server_key_iv: "be86e90396cd198f30dd9e5e67fde591"
server_key_version: "encryptionkey@0"
I20260929 02:55:18.361809 17849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:18.383569 17849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:18.387475 17849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:18.387751 17849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:18.403980 17849 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:46691
I20260929 02:55:18.404012 17915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:46691 every 8 connection(s)
I20260929 02:55:18.405313 17849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:18.408913 17916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:18.411996 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 17849
I20260929 02:55:18.412195 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:18.412746 15433 external_mini_cluster.cc:1428] Setting key cb2ea8279a01a34ee19c27d491f0381b
I20260929 02:55:18.421073 17916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738: Bootstrap starting.
I20260929 02:55:18.424999 17916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:18.426334 17916 log.cc:824] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:55:18 dist-test-slave-q5f0 krb5kdc[17836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:18.429265 17916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738: No bootstrap required, opened a new log
I20260929 02:55:18.435096 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:18.414608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.15.18.126:46691)
0929 02:55:18.415334 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:18.415346 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:18.415385 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:55:18.417010 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:18.417034 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:18.417469 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:18.417809 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:18.419598 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:18.420919 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:18.422686 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:18.423498 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:18.423664 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:18.428117 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:18.428125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:18.428141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:18.428179 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:18.431083 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:18.431820 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:18.431826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:18.431833 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:18.432004 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:18.432363 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:18.432366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:18.432369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:18.432901 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:18.433202 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:18.433636 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 02:55:18.434058 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":336,"threads_started":1}
I20260929 02:55:18.437158 17916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626c55995b06447b9c7120b13eefd738" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46691 } }
I20260929 02:55:18.437587 17916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:18.437686 17916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626c55995b06447b9c7120b13eefd738, State: Initialized, Role: FOLLOWER
I20260929 02:55:18.438283 17916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626c55995b06447b9c7120b13eefd738" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46691 } }
I20260929 02:55:18.438510 17916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:18.438603 17916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:18.438722 17916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:18.440954 17916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626c55995b06447b9c7120b13eefd738" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46691 } }
I20260929 02:55:18.441498 17916 leader_election.cc:302] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 626c55995b06447b9c7120b13eefd738; no voters:
I20260929 02:55:18.442059 17916 leader_election.cc:288] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:18.442189 17921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:18.442778 17921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [term 1 LEADER]: Becoming Leader. State: Replica: 626c55995b06447b9c7120b13eefd738, State: Running, Role: LEADER
I20260929 02:55:18.443259 17921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626c55995b06447b9c7120b13eefd738" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46691 } }
I20260929 02:55:18.443990 17916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:18.445952 17923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 626c55995b06447b9c7120b13eefd738. Latest consensus state: current_term: 1 leader_uuid: "626c55995b06447b9c7120b13eefd738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626c55995b06447b9c7120b13eefd738" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46691 } } }
I20260929 02:55:18.446542 17923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:18.445932 17922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "626c55995b06447b9c7120b13eefd738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626c55995b06447b9c7120b13eefd738" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46691 } } }
I20260929 02:55:18.448833 17922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:18.449064 17928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:18.453712 17928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:18.462673 17928 catalog_manager.cc:1347] Generated new cluster ID: ba998850c53c4f45aedbaf398e80974f
I20260929 02:55:18.462785 17928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:18.475565 17928 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:18.476876 17928 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:18.487102 17928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 626c55995b06447b9c7120b13eefd738: Generated new TSK 0
I20260929 02:55:18.488001 17928 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:55:18.531611 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46691
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:18.850996 17942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:18.851274 17942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:18.851404 17942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:18.858600 17942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:18.858714 17942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:18.858858 17942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:18.868855 17942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:18.870148 17942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:18.871436 17942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:18.884002 17948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:18.884089 17950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:18.884024 17947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:18.884624 17942 server_base.cc:1034] running on GCE node
I20260929 02:55:18.885257 17942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:18.886274 17942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:18.887533 17942 hybrid_clock.cc:648] HybridClock initialized: now 1790650518887445 us; error 55 us; skew 500 ppm
I20260929 02:55:18.887843 17942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:18 dist-test-slave-q5f0 krb5kdc[17836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:18.890883 17942 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:55:18.892299 17942 webserver.cc:460] Webserver started at http://127.15.18.65:43017/ using document root <none> and password file <none>
I20260929 02:55:18.892944 17942 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:18.893057 17942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:18.893328 17942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:18.896240 17942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "4257eced7a584958928e62200b3aa116"
format_stamp: "Formatted at 2026-09-29 02:55:18 on dist-test-slave-q5f0"
server_key: "548a165fc951294e17ef62fd69d6f8d5"
server_key_iv: "4ef9a2d8329d809513bf4cd1bc4417ee"
server_key_version: "encryptionkey@0"
I20260929 02:55:18.896982 17942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "4257eced7a584958928e62200b3aa116"
format_stamp: "Formatted at 2026-09-29 02:55:18 on dist-test-slave-q5f0"
server_key: "548a165fc951294e17ef62fd69d6f8d5"
server_key_iv: "4ef9a2d8329d809513bf4cd1bc4417ee"
server_key_version: "encryptionkey@0"
I20260929 02:55:18.902099 17942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:55:18.905567 17957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:18.906927 17942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:55:18.907086 17942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "4257eced7a584958928e62200b3aa116"
format_stamp: "Formatted at 2026-09-29 02:55:18 on dist-test-slave-q5f0"
server_key: "548a165fc951294e17ef62fd69d6f8d5"
server_key_iv: "4ef9a2d8329d809513bf4cd1bc4417ee"
server_key_version: "encryptionkey@0"
I20260929 02:55:18.907275 17942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:18.919346 17942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:18.923385 17942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:18.923712 17942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:18.924683 17942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:18.926052 17942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:18.926136 17942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:18.926254 17942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:18.926308 17942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:18.955848 17942 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:43419
I20260929 02:55:18.955893 18069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:43419 every 8 connection(s)
I20260929 02:55:18.957063 17942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:18.966017 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 17942
I20260929 02:55:18.966228 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:18.966988 15433 external_mini_cluster.cc:1428] Setting key 7ea03c75e37b03643dc548d743fcd2ff
Sep 29 02:55:18 dist-test-slave-q5f0 krb5kdc[17836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:18.976363 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:18.959361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:33889 (local address 127.15.18.126:46691)
0929 02:55:18.959800 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:18.959806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:18.960710 (+ 904us) server_negotiation.cc:407] Connection header received
0929 02:55:18.962451 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:18.962456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:18.962530 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:18.962650 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:18.964177 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:18.964888 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:18.966867 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:18.967297 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:18.967400 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:18.970929 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:18.970934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:18.970937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:18.970971 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:18.973342 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:18.974013 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:18.974017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:18.974019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:18.974155 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:18.974658 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:18.974662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:18.974665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:18.974998 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:18.975133 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:18.975911 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 02:55:18.976101 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":212,"threads_started":1}
I20260929 02:55:18.977707 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:18.959900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46691 (local address 127.15.18.65:33889)
0929 02:55:18.960558 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:18.960595 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:18.962185 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:18.962854 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:18.962862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:18.963432 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:18.964023 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:18.964040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:18.965029 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:18.965033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:18.966676 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:18.966691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:18.967538 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:18.967547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:18.967745 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:18.968383 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:18.968411 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:18.970726 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:18.973468 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:18.973477 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:18.973502 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:18.973890 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:18.974282 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:18.974287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:18.974291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:18.974545 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:18.975108 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:18.975114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:18.975538 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 02:55:18.975825 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 02:55:18.976350 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":242,"threads_started":1}
I20260929 02:55:18.979216 18070 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46691
I20260929 02:55:18.979491 18070 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:18.980216 18070 heartbeater.cc:511] Master 127.15.18.126:46691 requested a full tablet report, sending...
I20260929 02:55:18.982393 17881 ts_manager.cc:194] Registered new tserver with Master: 4257eced7a584958928e62200b3aa116 (127.15.18.65:43419)
I20260929 02:55:18.984443 17881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:33889
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:55:19.010591 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46691
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:19.329907 18076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:19.330193 18076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:19.330314 18076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:19.337705 18076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:19.337822 18076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:19.337970 18076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:19.347707 18076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:19.349053 18076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:19.350396 18076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:19.363046 18082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:19.363114 18081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:19.363091 18084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:19.363955 18076 server_base.cc:1034] running on GCE node
I20260929 02:55:19.364638 18076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:19.365708 18076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:19.366941 18076 hybrid_clock.cc:648] HybridClock initialized: now 1790650519366891 us; error 45 us; skew 500 ppm
I20260929 02:55:19.367257 18076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:19 dist-test-slave-q5f0 krb5kdc[17836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650519, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:19.370577 18076 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:55:19.372444 18076 webserver.cc:460] Webserver started at http://127.15.18.66:33231/ using document root <none> and password file <none>
I20260929 02:55:19.373332 18076 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:19.373461 18076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:19.373803 18076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:19.376904 18076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "e737f11acd6f4da7a6ed8d55671944f8"
format_stamp: "Formatted at 2026-09-29 02:55:19 on dist-test-slave-q5f0"
server_key: "2ced305cffe175c8002a29d697225eae"
server_key_iv: "2edf35dba63fbfb22177a4e7e511ebc5"
server_key_version: "encryptionkey@0"
I20260929 02:55:19.377629 18076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "e737f11acd6f4da7a6ed8d55671944f8"
format_stamp: "Formatted at 2026-09-29 02:55:19 on dist-test-slave-q5f0"
server_key: "2ced305cffe175c8002a29d697225eae"
server_key_iv: "2edf35dba63fbfb22177a4e7e511ebc5"
server_key_version: "encryptionkey@0"
I20260929 02:55:19.382769 18076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:19.386286 18091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:19.387748 18076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:19.387908 18076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "e737f11acd6f4da7a6ed8d55671944f8"
format_stamp: "Formatted at 2026-09-29 02:55:19 on dist-test-slave-q5f0"
server_key: "2ced305cffe175c8002a29d697225eae"
server_key_iv: "2edf35dba63fbfb22177a4e7e511ebc5"
server_key_version: "encryptionkey@0"
I20260929 02:55:19.388072 18076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:19.421425 18076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:19.426615 18076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:19.426964 18076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:19.428241 18076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:19.429929 18076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:19.430001 18076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:19.430094 18076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:19.430142 18076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:19.459175 18076 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:42981
I20260929 02:55:19.459210 18203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:42981 every 8 connection(s)
I20260929 02:55:19.460465 18076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:19.468065 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 18076
I20260929 02:55:19.468237 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:19.468634 15433 external_mini_cluster.cc:1428] Setting key 06c71a76d5cb5fe22a0003fcbd087484
Sep 29 02:55:19 dist-test-slave-q5f0 krb5kdc[17836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650519, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:19.480803 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:19.462977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:33055 (local address 127.15.18.126:46691)
0929 02:55:19.463174 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:19.463178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:19.464437 (+ 1259us) server_negotiation.cc:407] Connection header received
0929 02:55:19.466182 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:19.466188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:19.466277 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:19.466361 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:19.467911 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:19.468701 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:19.470366 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:19.470921 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:19.471048 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:19.474747 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:19.474754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:19.474759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:19.474800 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:19.477556 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:19.478449 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:19.478453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:19.478455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:19.478587 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:19.479045 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:19.479048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:19.479050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:19.479311 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:19.479447 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:19.480252 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 02:55:19.480480 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:55:19.481892 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:19.463467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46691 (local address 127.15.18.66:33055)
0929 02:55:19.464267 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:19.464325 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:19.465943 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:19.466571 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:19.466579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:19.467087 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:19.467742 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:19.467762 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:19.468843 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:19.468847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:19.470227 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:19.470242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:19.471130 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:19.471135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:19.471373 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:19.472165 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:19.472194 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:19.474502 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:19.477713 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:19.477723 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:19.477741 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:19.478333 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:19.478690 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:19.478694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:19.478698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:19.478952 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:19.479470 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:19.479479 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:19.479869 (+ 390us) client_negotiation.cc:769] Sending connection context
0929 02:55:19.480133 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 02:55:19.480708 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":332,"threads_started":1}
I20260929 02:55:19.483338 18204 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46691
I20260929 02:55:19.483646 18204 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:19.484344 18204 heartbeater.cc:511] Master 127.15.18.126:46691 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
I20260929 02:55:19.485755 17881 ts_manager.cc:194] Registered new tserver with Master: e737f11acd6f4da7a6ed8d55671944f8 (127.15.18.66:42981)
I20260929 02:55:19.486553 17881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:33055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:55:19.508894 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46691
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:19.825850 18209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:19.826143 18209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:19.826267 18209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:19.833678 18209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:19.833822 18209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:19.833979 18209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:19.844013 18209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:19.845398 18209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:19.846695 18209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:19.859738 18214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:19.859714 18215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:19.859973 18217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:19.860690 18209 server_base.cc:1034] running on GCE node
I20260929 02:55:19.861348 18209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:19.862303 18209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:19.863559 18209 hybrid_clock.cc:648] HybridClock initialized: now 1790650519863483 us; error 37 us; skew 500 ppm
I20260929 02:55:19.863901 18209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:19 dist-test-slave-q5f0 krb5kdc[17836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650519, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:19.871793 18209 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:55:19.873369 18209 webserver.cc:460] Webserver started at http://127.15.18.67:40979/ using document root <none> and password file <none>
I20260929 02:55:19.874095 18209 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:19.874214 18209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:19.874500 18209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:19.877537 18209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "b91b58d3257842e1a899022749614e40"
format_stamp: "Formatted at 2026-09-29 02:55:19 on dist-test-slave-q5f0"
server_key: "88558ec699b67a3f8fae10d0458276c8"
server_key_iv: "cf8fe7183fd745631e08e3e5e776f9d2"
server_key_version: "encryptionkey@0"
I20260929 02:55:19.878350 18209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "b91b58d3257842e1a899022749614e40"
format_stamp: "Formatted at 2026-09-29 02:55:19 on dist-test-slave-q5f0"
server_key: "88558ec699b67a3f8fae10d0458276c8"
server_key_iv: "cf8fe7183fd745631e08e3e5e776f9d2"
server_key_version: "encryptionkey@0"
I20260929 02:55:19.883677 18209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:55:19.887223 18224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:19.888798 18209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:19.888969 18209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "b91b58d3257842e1a899022749614e40"
format_stamp: "Formatted at 2026-09-29 02:55:19 on dist-test-slave-q5f0"
server_key: "88558ec699b67a3f8fae10d0458276c8"
server_key_iv: "cf8fe7183fd745631e08e3e5e776f9d2"
server_key_version: "encryptionkey@0"
I20260929 02:55:19.889145 18209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:19.904567 18209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:19.908737 18209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:19.908989 18209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:19.909933 18209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:19.911307 18209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:19.911379 18209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:19.911464 18209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:19.911527 18209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:19.941366 18209 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:41015
I20260929 02:55:19.941403 18336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:41015 every 8 connection(s)
I20260929 02:55:19.942628 18209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:19.946265 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 18209
I20260929 02:55:19.946458 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:19.946915 15433 external_mini_cluster.cc:1428] Setting key a27fa4ecb39c5015a5843afa6fa85ce2
Sep 29 02:55:19 dist-test-slave-q5f0 krb5kdc[17836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650519, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:19.963148 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:19.945266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:41189 (local address 127.15.18.126:46691)
0929 02:55:19.945444 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:19.945449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:19.946744 (+ 1295us) server_negotiation.cc:407] Connection header received
0929 02:55:19.948855 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:19.948861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:19.948973 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:19.949074 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:19.950674 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:19.951457 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:19.953024 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:19.953427 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:19.953522 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:19.957229 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:19.957235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:19.957238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:19.957272 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:19.959883 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:19.960734 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:19.960741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:19.960745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:19.960912 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:19.961431 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:19.961436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:19.961439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:19.961809 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:19.961946 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:19.962725 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 02:55:19.962915 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:55:19.964241 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:19.945763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46691 (local address 127.15.18.67:41189)
0929 02:55:19.946569 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:19.946614 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:19.948564 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:19.949300 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:19.949308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:19.949928 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:19.950511 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:19.950529 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:19.951624 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:19.951628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:19.952886 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:19.952898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:19.953595 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:19.953599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:19.953792 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:19.954416 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:19.954439 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:19.956993 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:19.960028 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:19.960038 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:19.960054 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:19.960600 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:19.961065 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:19.961069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:19.961072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:19.961316 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:19.961971 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:19.961979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:19.962405 (+ 426us) client_negotiation.cc:769] Sending connection context
0929 02:55:19.962641 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 02:55:19.963037 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":312,"threads_started":1}
I20260929 02:55:19.965997 18337 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46691
I20260929 02:55:19.966315 18337 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:19.967047 18337 heartbeater.cc:511] Master 127.15.18.126:46691 requested a full tablet report, sending...
I20260929 02:55:19.968591 17881 ts_manager.cc:194] Registered new tserver with Master: b91b58d3257842e1a899022749614e40 (127.15.18.67:41015)
I20260929 02:55:19.969405 17881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:41189
I20260929 02:55:19.974840 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:55:19.987927 18070 heartbeater.cc:503] Master 127.15.18.126:46691 was elected leader, sending a full tablet report...
W20260929 02:55:20.003361 18072 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:55:20.001006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.15.18.126:46691)
0929 02:55:20.001179 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:20.001184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:20.001614 (+ 430us) server_negotiation.cc:407] Connection header received
0929 02:55:20.001749 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:20.001753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:20.001820 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:20.001913 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:20.003092 (+ 1179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32928: BlockingRecv error: recv got EOF from 127.0.0.1:32928 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:55:20.006927 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 17942
W20260929 02:55:20.009639 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 17942: Not authorized: Client connection negotiation failed: client connection to 127.15.18.65:43419: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:55:20.009789 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 17942
I20260929 02:55:20.022240 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18076
W20260929 02:55:20.025131 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18076: Not authorized: Client connection negotiation failed: client connection to 127.15.18.66:42981: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:55:20.025305 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 18076
I20260929 02:55:20.037271 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18209
W20260929 02:55:20.040202 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18209: Not authorized: Client connection negotiation failed: client connection to 127.15.18.67:41015: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:55:20.040359 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 18209
I20260929 02:55:20.052254 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 17849
I20260929 02:55:20.053654 17911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:20.155308 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 17849
2026-09-29T02:55:20Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3339 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:20 dist-test-slave-q5f0 krb5kdc[18354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:20 dist-test-slave-q5f0 krb5kdc[18354](info): set up 2 sockets
Sep 29 02:55:20 dist-test-slave-q5f0 krb5kdc[18354](info): commencing operation
krb5kdc: starting...
W20260929 02:55:21.217849 15433 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 02:55:21.428508 15433 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.210s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 02:55:21.660595 15433 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.232s user 0.001s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 02:55:21.944639 15433 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.284s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/krb5kdc/test-user.keytab.
W20260929 02:55:22.628275 15433 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.683s user 0.000s sys 0.004s
Sep 29 02:55:22 dist-test-slave-q5f0 krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:55:22.776070 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:39481
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:41933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:39481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:23.088563 18367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:23.088896 18367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:23.089039 18367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:23.096282 18367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:23.096422 18367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:23.096496 18367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:23.096572 18367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:23.096639 18367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:23.106995 18367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:41933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:39481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:39481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:23.108381 18367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:23.109710 18367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:23.121124 18373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:23.121155 18372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:23.121183 18375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:23.121930 18367 server_base.cc:1034] running on GCE node
I20260929 02:55:23.122685 18367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:23.124115 18367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:23.125327 18367 hybrid_clock.cc:648] HybridClock initialized: now 1790650523125283 us; error 31 us; skew 500 ppm
I20260929 02:55:23.125648 18367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:23 dist-test-slave-q5f0 krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650523, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:23.128783 18367 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:55:23.130235 18367 webserver.cc:460] Webserver started at http://127.15.18.126:37821/ using document root <none> and password file <none>
I20260929 02:55:23.130949 18367 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:23.131057 18367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:23.131348 18367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:23.134397 18367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "b878bc2094df4cf799dd1f2dd8d4784f"
format_stamp: "Formatted at 2026-09-29 02:55:23 on dist-test-slave-q5f0"
server_key: "bfc10025a7004ff7118ca8e0b069bb47"
server_key_iv: "67aeb6dc4e40ab5542f2e199b6344007"
server_key_version: "encryptionkey@0"
I20260929 02:55:23.135174 18367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "b878bc2094df4cf799dd1f2dd8d4784f"
format_stamp: "Formatted at 2026-09-29 02:55:23 on dist-test-slave-q5f0"
server_key: "bfc10025a7004ff7118ca8e0b069bb47"
server_key_iv: "67aeb6dc4e40ab5542f2e199b6344007"
server_key_version: "encryptionkey@0"
I20260929 02:55:23.140579 18367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:55:23.144366 18382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:23.145846 18367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:23.146054 18367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "b878bc2094df4cf799dd1f2dd8d4784f"
format_stamp: "Formatted at 2026-09-29 02:55:23 on dist-test-slave-q5f0"
server_key: "bfc10025a7004ff7118ca8e0b069bb47"
server_key_iv: "67aeb6dc4e40ab5542f2e199b6344007"
server_key_version: "encryptionkey@0"
I20260929 02:55:23.146246 18367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:23.166623 18367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:23.170943 18367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:23.171221 18367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:23.188584 18367 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:39481
I20260929 02:55:23.188625 18433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:39481 every 8 connection(s)
I20260929 02:55:23.190001 18367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:23.193899 18434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:23.194195 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 18367
I20260929 02:55:23.194368 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:23.194977 15433 external_mini_cluster.cc:1428] Setting key 95eb2a0f8d2a65dd3ba682ca9a43916d
I20260929 02:55:23.208352 18434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f: Bootstrap starting.
Sep 29 02:55:23 dist-test-slave-q5f0 krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:23.211743 18434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:23.213105 18434 log.cc:824] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:23.216661 18434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f: No bootstrap required, opened a new log
I20260929 02:55:23.220615 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:23.197036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.15.18.126:39481)
0929 02:55:23.197862 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:23.197878 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:23.197931 (+ 53us) server_negotiation.cc:407] Connection header received
0929 02:55:23.200364 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:23.200399 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:23.201045 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:23.201566 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:23.202996 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:23.205008 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:23.206395 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:23.207467 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:23.207692 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:23.211867 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:23.211878 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:23.211900 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:23.211954 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:23.215478 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:23.216316 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:23.216324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:23.216333 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:23.216550 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:23.217017 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:23.217022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:23.217025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:23.217796 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:23.218075 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:23.218493 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 02:55:23.218987 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":335,"threads_started":1}
I20260929 02:55:23.225069 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:55:23.226323 18434 raft_consensus.cc:348] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b878bc2094df4cf799dd1f2dd8d4784f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39481 } }
I20260929 02:55:23.226701 18434 raft_consensus.cc:374] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:23.226799 18434 raft_consensus.cc:729] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b878bc2094df4cf799dd1f2dd8d4784f, State: Initialized, Role: FOLLOWER
I20260929 02:55:23.227377 18434 consensus_queue.cc:260] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b878bc2094df4cf799dd1f2dd8d4784f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39481 } }
I20260929 02:55:23.227645 18434 raft_consensus.cc:388] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:23.227779 18434 raft_consensus.cc:482] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:23.227934 18434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:23.230082 18434 raft_consensus.cc:504] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b878bc2094df4cf799dd1f2dd8d4784f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39481 } }
I20260929 02:55:23.230590 18434 leader_election.cc:302] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b878bc2094df4cf799dd1f2dd8d4784f; no voters:
I20260929 02:55:23.231197 18434 leader_election.cc:288] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:23.231292 18440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:23.232406 18440 raft_consensus.cc:686] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [term 1 LEADER]: Becoming Leader. State: Replica: b878bc2094df4cf799dd1f2dd8d4784f, State: Running, Role: LEADER
I20260929 02:55:23.232963 18440 consensus_queue.cc:237] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b878bc2094df4cf799dd1f2dd8d4784f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39481 } }
I20260929 02:55:23.233559 18434 sys_catalog.cc:564] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:23.235669 18442 sys_catalog.cc:455] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b878bc2094df4cf799dd1f2dd8d4784f. Latest consensus state: current_term: 1 leader_uuid: "b878bc2094df4cf799dd1f2dd8d4784f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b878bc2094df4cf799dd1f2dd8d4784f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39481 } } }
I20260929 02:55:23.235854 18441 sys_catalog.cc:455] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b878bc2094df4cf799dd1f2dd8d4784f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b878bc2094df4cf799dd1f2dd8d4784f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39481 } } }
I20260929 02:55:23.236171 18441 sys_catalog.cc:458] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:23.236070 18442 sys_catalog.cc:458] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:23.243496 18448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:23.249094 18448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:23.270577 18448 catalog_manager.cc:1347] Generated new cluster ID: 5a4b7bf8d33647d3a269ffe8fa6e75ee
I20260929 02:55:23.270718 18448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:23.281847 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:23.273081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.15.18.126:39481)
0929 02:55:23.273238 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:23.273244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:23.273755 (+ 511us) server_negotiation.cc:407] Connection header received
0929 02:55:23.273923 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:23.273929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:23.274018 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:23.274139 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:23.275250 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:23.276448 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:23.278145 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:23.278820 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:23.278958 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:23.280173 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:23.280180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:23.280185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:23.281045 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:23.281120 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:23.281153 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:23.281262 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:23.281367 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 02:55:23.281465 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":51}
I20260929 02:55:23.298568 18448 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:23.300480 18448 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:23.310529 18448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b878bc2094df4cf799dd1f2dd8d4784f: Generated new TSK 0
I20260929 02:55:23.311647 18448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:55:23.323068 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18367
I20260929 02:55:23.324533 18429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:23.417032 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 18367
2026-09-29T02:55:23Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3260 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:23 dist-test-slave-q5f0 krb5kdc[18468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:23 dist-test-slave-q5f0 krb5kdc[18468](info): set up 2 sockets
Sep 29 02:55:23 dist-test-slave-q5f0 krb5kdc[18468](info): commencing operation
krb5kdc: starting...
W20260929 02:55:24.415936 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:55:24 dist-test-slave-q5f0 krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:55:24.600329 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46591
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:46591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:24.913751 18481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:24.914058 18481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:24.914172 18481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:24.921430 18481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:24.921545 18481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:24.921603 18481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:24.921684 18481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:24.921741 18481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:24.931701 18481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:46591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:24.933044 18481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:24.934358 18481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:24.945516 18489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:24.945545 18487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:24.945577 18486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:24.946312 18481 server_base.cc:1034] running on GCE node
I20260929 02:55:24.947057 18481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:24.948452 18481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:24.949656 18481 hybrid_clock.cc:648] HybridClock initialized: now 1790650524949608 us; error 43 us; skew 500 ppm
I20260929 02:55:24.949996 18481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:24 dist-test-slave-q5f0 krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650524, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:24.956600 18481 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:55:24.958012 18481 webserver.cc:460] Webserver started at http://127.15.18.126:34901/ using document root <none> and password file <none>
I20260929 02:55:24.958660 18481 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:24.958766 18481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:24.959054 18481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:24.961961 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "f3dc2bafd9c7446cae612a209ce91fd8"
format_stamp: "Formatted at 2026-09-29 02:55:24 on dist-test-slave-q5f0"
server_key: "5950cfad4a77404d666898e7b469c466"
server_key_iv: "8463b851dc10ecc65657be0335ff0f8a"
server_key_version: "encryptionkey@0"
I20260929 02:55:24.962667 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "f3dc2bafd9c7446cae612a209ce91fd8"
format_stamp: "Formatted at 2026-09-29 02:55:24 on dist-test-slave-q5f0"
server_key: "5950cfad4a77404d666898e7b469c466"
server_key_iv: "8463b851dc10ecc65657be0335ff0f8a"
server_key_version: "encryptionkey@0"
I20260929 02:55:24.967757 18481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 02:55:24.970974 18496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:24.972295 18481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:24.972448 18481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "f3dc2bafd9c7446cae612a209ce91fd8"
format_stamp: "Formatted at 2026-09-29 02:55:24 on dist-test-slave-q5f0"
server_key: "5950cfad4a77404d666898e7b469c466"
server_key_iv: "8463b851dc10ecc65657be0335ff0f8a"
server_key_version: "encryptionkey@0"
I20260929 02:55:24.972618 18481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:24.999079 18481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:25.003424 18481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:25.003736 18481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:25.021457 18481 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:46591
I20260929 02:55:25.021478 18547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:46591 every 8 connection(s)
I20260929 02:55:25.022872 18481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:25.026774 18548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:25.028007 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 18481
I20260929 02:55:25.028201 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:25.028972 15433 external_mini_cluster.cc:1428] Setting key 737ae587605d6a674c42b2cd9e43ee4c
I20260929 02:55:25.040853 18548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8: Bootstrap starting.
Sep 29 02:55:25 dist-test-slave-q5f0 krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:25.044560 18548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:25.045600 18548 log.cc:824] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:25.048804 18548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8: No bootstrap required, opened a new log
I20260929 02:55:25.051848 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:25.031098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.15.18.126:46591)
0929 02:55:25.031934 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:25.031950 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:25.031993 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:55:25.033638 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:25.033665 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:25.034081 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:25.034405 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:25.036027 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:25.037286 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:25.039015 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:25.039733 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:25.039863 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:25.044275 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:25.044282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:25.044296 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:25.044339 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:25.047574 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:25.048398 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:25.048405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:25.048411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:25.048593 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:25.049011 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:25.049015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:25.049017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:25.049565 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:25.049795 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:25.050241 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 02:55:25.050613 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":435,"threads_started":1}
I20260929 02:55:25.056813 18548 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3dc2bafd9c7446cae612a209ce91fd8" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46591 } }
I20260929 02:55:25.057164 18548 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:25.057262 18548 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3dc2bafd9c7446cae612a209ce91fd8, State: Initialized, Role: FOLLOWER
I20260929 02:55:25.057821 18548 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3dc2bafd9c7446cae612a209ce91fd8" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46591 } }
I20260929 02:55:25.058017 18548 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:25.058135 18548 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:25.058276 18548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:25.060410 18548 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3dc2bafd9c7446cae612a209ce91fd8" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46591 } }
I20260929 02:55:25.060868 18548 leader_election.cc:302] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3dc2bafd9c7446cae612a209ce91fd8; no voters:
I20260929 02:55:25.061437 18548 leader_election.cc:288] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:25.061545 18553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:25.062606 18553 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [term 1 LEADER]: Becoming Leader. State: Replica: f3dc2bafd9c7446cae612a209ce91fd8, State: Running, Role: LEADER
I20260929 02:55:25.063208 18553 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3dc2bafd9c7446cae612a209ce91fd8" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46591 } }
I20260929 02:55:25.063536 18548 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:25.066118 18554 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3dc2bafd9c7446cae612a209ce91fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3dc2bafd9c7446cae612a209ce91fd8" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46591 } } }
I20260929 02:55:25.066149 18555 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3dc2bafd9c7446cae612a209ce91fd8. Latest consensus state: current_term: 1 leader_uuid: "f3dc2bafd9c7446cae612a209ce91fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3dc2bafd9c7446cae612a209ce91fd8" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46591 } } }
I20260929 02:55:25.066641 18554 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:25.066641 18555 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:25.067487 18561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:25.072618 18561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:25.082775 18561 catalog_manager.cc:1347] Generated new cluster ID: e99de1ff8f1e48118cb9ae41b1c8169d
I20260929 02:55:25.082896 18561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:25.113562 18561 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:25.114918 18561 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:25.121773 18561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3dc2bafd9c7446cae612a209ce91fd8: Generated new TSK 0
I20260929 02:55:25.122598 18561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:55:25.180574 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46591
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:25.519167 18574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:25.519438 18574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:25.519546 18574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:25.526583 18574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:25.526693 18574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:25.526806 18574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:25.536350 18574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:25.537576 18574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:25.538789 18574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:25.551179 18579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:25.551216 18580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:25.551811 18582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:25.552683 18574 server_base.cc:1034] running on GCE node
I20260929 02:55:25.553308 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:25.554266 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:25.555475 18574 hybrid_clock.cc:648] HybridClock initialized: now 1790650525555418 us; error 50 us; skew 500 ppm
I20260929 02:55:25.555811 18574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:25 dist-test-slave-q5f0 krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650525, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:25.558797 18574 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:55:25.560330 18574 webserver.cc:460] Webserver started at http://127.15.18.65:35425/ using document root <none> and password file <none>
I20260929 02:55:25.560992 18574 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:25.561090 18574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:25.561334 18574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:25.564199 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "869ba6e5a1754587a200bed9e11a7dd7"
format_stamp: "Formatted at 2026-09-29 02:55:25 on dist-test-slave-q5f0"
server_key: "64ffe8f4257f6bad194ab0e239d82ed3"
server_key_iv: "cd13b12c9a14ca0644def6e0ce4c501e"
server_key_version: "encryptionkey@0"
I20260929 02:55:25.564939 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "869ba6e5a1754587a200bed9e11a7dd7"
format_stamp: "Formatted at 2026-09-29 02:55:25 on dist-test-slave-q5f0"
server_key: "64ffe8f4257f6bad194ab0e239d82ed3"
server_key_iv: "cd13b12c9a14ca0644def6e0ce4c501e"
server_key_version: "encryptionkey@0"
I20260929 02:55:25.570178 18574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:55:25.573885 18589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:25.575337 18574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:55:25.575489 18574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "869ba6e5a1754587a200bed9e11a7dd7"
format_stamp: "Formatted at 2026-09-29 02:55:25 on dist-test-slave-q5f0"
server_key: "64ffe8f4257f6bad194ab0e239d82ed3"
server_key_iv: "cd13b12c9a14ca0644def6e0ce4c501e"
server_key_version: "encryptionkey@0"
I20260929 02:55:25.575711 18574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:25.589882 18574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:25.594137 18574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:25.594406 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:25.595335 18574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:25.596731 18574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:25.596805 18574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:25.596887 18574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:25.596932 18574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:25.626994 18574 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:44511
I20260929 02:55:25.627065 18701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:44511 every 8 connection(s)
I20260929 02:55:25.628288 18574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:25.630812 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 18574
I20260929 02:55:25.631031 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:25.631757 15433 external_mini_cluster.cc:1428] Setting key 4ed5c2de0f55418733609ac813f204f9
Sep 29 02:55:25 dist-test-slave-q5f0 krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650525, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
I20260929 02:55:25.649924 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:25.630680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:57977 (local address 127.15.18.126:46591)
0929 02:55:25.631147 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:25.631153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:25.632218 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 02:55:25.634337 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:25.634341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:25.634409 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:25.634495 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:25.636498 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:25.637219 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:25.639310 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:25.639928 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:25.640082 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:25.643438 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:25.643444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:25.643449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:25.643485 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:25.646718 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:25.647547 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:25.647553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:25.647557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:25.647701 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:25.648134 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:25.648139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:25.648142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:25.648492 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:25.648635 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:25.649336 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 02:55:25.649567 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":219,"threads_started":1}
I20260929 02:55:25.650827 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:25.631231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46591 (local address 127.15.18.65:57977)
0929 02:55:25.632050 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:25.632096 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:25.634057 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:25.634747 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:25.634757 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:25.635460 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:25.636318 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:25.636341 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:25.637367 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:25.637372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:25.639163 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:25.639178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:25.640199 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:25.640204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:25.640432 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:25.641081 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:25.641105 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:25.643205 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:25.646890 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:25.646898 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:25.646913 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:25.647396 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:25.647817 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:25.647821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:25.647823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:25.648025 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:25.648605 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:25.648611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:25.648971 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 02:55:25.649193 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 02:55:25.649663 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":298,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
I20260929 02:55:25.652474 18702 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46591
I20260929 02:55:25.652801 18702 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:25.653683 18702 heartbeater.cc:511] Master 127.15.18.126:46591 requested a full tablet report, sending...
I20260929 02:55:25.655989 18513 ts_manager.cc:194] Registered new tserver with Master: 869ba6e5a1754587a200bed9e11a7dd7 (127.15.18.65:44511)
I20260929 02:55:25.657416 18513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:57977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:55:25.674595 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46591
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:25.986440 18708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:25.986753 18708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:25.986884 18708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:25.994261 18708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:25.994400 18708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:25.994563 18708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:26.004418 18708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:26.005764 18708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:26.007093 18708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:26.019771 18713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:26.019793 18716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:26.019748 18714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:26.020663 18708 server_base.cc:1034] running on GCE node
I20260929 02:55:26.021343 18708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:26.022382 18708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:26.023645 18708 hybrid_clock.cc:648] HybridClock initialized: now 1790650526023580 us; error 60 us; skew 500 ppm
I20260929 02:55:26.023998 18708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650526, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:26.027416 18708 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:55:26.029094 18708 webserver.cc:460] Webserver started at http://127.15.18.66:36367/ using document root <none> and password file <none>
I20260929 02:55:26.029806 18708 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:26.029927 18708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:26.030237 18708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:26.033365 18708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "844900e0c76f4892b1d11de8ca822f59"
format_stamp: "Formatted at 2026-09-29 02:55:26 on dist-test-slave-q5f0"
server_key: "0a163e6bb907ea708ddf8afc012f1523"
server_key_iv: "b92900bb7e44e32adf4ba3a44f2cfc27"
server_key_version: "encryptionkey@0"
I20260929 02:55:26.034152 18708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "844900e0c76f4892b1d11de8ca822f59"
format_stamp: "Formatted at 2026-09-29 02:55:26 on dist-test-slave-q5f0"
server_key: "0a163e6bb907ea708ddf8afc012f1523"
server_key_iv: "b92900bb7e44e32adf4ba3a44f2cfc27"
server_key_version: "encryptionkey@0"
I20260929 02:55:26.039592 18708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:55:26.043090 18723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:26.044492 18708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:55:26.044636 18708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "844900e0c76f4892b1d11de8ca822f59"
format_stamp: "Formatted at 2026-09-29 02:55:26 on dist-test-slave-q5f0"
server_key: "0a163e6bb907ea708ddf8afc012f1523"
server_key_iv: "b92900bb7e44e32adf4ba3a44f2cfc27"
server_key_version: "encryptionkey@0"
I20260929 02:55:26.044816 18708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:26.076340 18708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:26.081162 18708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:26.081466 18708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:26.082496 18708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:26.083921 18708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:26.084004 18708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:26.084126 18708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:26.084180 18708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:26.114610 18708 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:38089
I20260929 02:55:26.114642 18835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:38089 every 8 connection(s)
I20260929 02:55:26.115944 18708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:26.122150 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 18708
I20260929 02:55:26.122341 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:26.122875 15433 external_mini_cluster.cc:1428] Setting key 203c1441932dc05aa7f5a0d62b053f09
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650526, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:26.137231 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:26.118460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:47901 (local address 127.15.18.126:46591)
0929 02:55:26.118630 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:26.118634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:26.119751 (+ 1117us) server_negotiation.cc:407] Connection header received
0929 02:55:26.121550 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:26.121555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:26.121629 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:26.121735 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:26.123717 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.124665 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.126832 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.127232 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.127325 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:26.131201 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:26.131207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:26.131211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:26.131249 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:26.134327 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.135020 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.135025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.135029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.135169 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.135607 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.135612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.135614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.135934 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:26.136069 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:26.136702 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 02:55:26.136922 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:55:26.138168 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:26.118909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46591 (local address 127.15.18.66:47901)
0929 02:55:26.119603 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:26.119646 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:26.121248 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:26.122031 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:26.122042 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:26.122733 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:26.123546 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:26.123571 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.124799 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.124803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:26.126685 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:26.126700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.127443 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.127449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:26.127752 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:26.128474 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:26.128502 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:26.130965 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:26.134474 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:26.134481 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:26.134495 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:26.134897 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:26.135274 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:26.135278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:26.135280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:26.135469 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:26.136037 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:26.136043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:26.136358 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 02:55:26.136578 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 02:55:26.136971 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":267,"threads_started":1}
I20260929 02:55:26.139608 18836 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46591
I20260929 02:55:26.139950 18836 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:26.140833 18836 heartbeater.cc:511] Master 127.15.18.126:46591 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
I20260929 02:55:26.142424 18513 ts_manager.cc:194] Registered new tserver with Master: 844900e0c76f4892b1d11de8ca822f59 (127.15.18.66:38089)
I20260929 02:55:26.143253 18513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:47901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:55:26.167320 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46591
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:26.490512 18841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:26.490856 18841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:26.491026 18841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:26.498520 18841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:26.498668 18841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:26.498843 18841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:26.509048 18841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:26.510421 18841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:26.511825 18841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:26.524988 18846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:26.524945 18849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:26.524920 18847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:26.526304 18841 server_base.cc:1034] running on GCE node
I20260929 02:55:26.526999 18841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:26.528050 18841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:26.529258 18841 hybrid_clock.cc:648] HybridClock initialized: now 1790650526529214 us; error 49 us; skew 500 ppm
I20260929 02:55:26.529585 18841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650526, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:26.532706 18841 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:55:26.534147 18841 webserver.cc:460] Webserver started at http://127.15.18.67:33451/ using document root <none> and password file <none>
I20260929 02:55:26.534847 18841 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:26.534972 18841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:26.535270 18841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:26.538280 18841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "19b1c756bd7f4e98bd4f8912d27cab6f"
format_stamp: "Formatted at 2026-09-29 02:55:26 on dist-test-slave-q5f0"
server_key: "c85b87b1263484284c76cc7ae2247a9d"
server_key_iv: "118181bbe538d2710e0233a3e77cf640"
server_key_version: "encryptionkey@0"
I20260929 02:55:26.539059 18841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "19b1c756bd7f4e98bd4f8912d27cab6f"
format_stamp: "Formatted at 2026-09-29 02:55:26 on dist-test-slave-q5f0"
server_key: "c85b87b1263484284c76cc7ae2247a9d"
server_key_iv: "118181bbe538d2710e0233a3e77cf640"
server_key_version: "encryptionkey@0"
I20260929 02:55:26.544426 18841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:55:26.548142 18856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:26.549603 18841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:55:26.549755 18841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "19b1c756bd7f4e98bd4f8912d27cab6f"
format_stamp: "Formatted at 2026-09-29 02:55:26 on dist-test-slave-q5f0"
server_key: "c85b87b1263484284c76cc7ae2247a9d"
server_key_iv: "118181bbe538d2710e0233a3e77cf640"
server_key_version: "encryptionkey@0"
I20260929 02:55:26.549965 18841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:26.583250 18841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:26.587390 18841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:26.587683 18841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:26.588701 18841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:26.590044 18841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:26.590119 18841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:26.590247 18841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:26.590301 18841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:26.619753 18841 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:41881
I20260929 02:55:26.619791 18968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:41881 every 8 connection(s)
I20260929 02:55:26.620991 18841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:26.625217 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 18841
I20260929 02:55:26.625355 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:26.625774 15433 external_mini_cluster.cc:1428] Setting key e271ad9b0c1eae02665ce650c80e50b7
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650526, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:26.639729 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:26.623345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:50303 (local address 127.15.18.126:46591)
0929 02:55:26.623501 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:26.623505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:26.624714 (+ 1209us) server_negotiation.cc:407] Connection header received
0929 02:55:26.626580 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:26.626586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:26.626709 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:26.626804 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:26.628491 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.629391 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.630919 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.631306 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.631410 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:26.634504 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:26.634509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:26.634512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:26.634543 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:26.636834 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.637531 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.637535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.637537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.637656 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.638055 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.638059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.638061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.638322 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:26.638450 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:26.639139 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 02:55:26.639351 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:55:26.640805 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:26.623858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46591 (local address 127.15.18.67:50303)
0929 02:55:26.624560 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:26.624597 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:26.626242 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:26.627032 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:26.627040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:26.627707 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:26.628326 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:26.628344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.629547 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.629551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:26.630779 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:26.630791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.631540 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.631548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:26.631757 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:26.632302 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:26.632327 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:26.634291 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:26.636977 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:26.636986 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:26.637003 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:26.637411 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:26.637757 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:26.637762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:26.637766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:26.637946 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:26.638444 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:26.638452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:26.638784 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 02:55:26.639018 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 02:55:26.639602 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":300,"threads_started":1}
I20260929 02:55:26.642035 18969 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46591
I20260929 02:55:26.642313 18969 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:26.643021 18969 heartbeater.cc:511] Master 127.15.18.126:46591 requested a full tablet report, sending...
I20260929 02:55:26.644382 18513 ts_manager.cc:194] Registered new tserver with Master: 19b1c756bd7f4e98bd4f8912d27cab6f (127.15.18.67:41881)
I20260929 02:55:26.645140 18513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:50303
I20260929 02:55:26.653784 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:55:26.660434 18702 heartbeater.cc:503] Master 127.15.18.126:46591 was elected leader, sending a full tablet report...
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650526, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650526, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:26.699483 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:26.685948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.15.18.126:46591)
0929 02:55:26.686160 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:26.686167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:26.686478 (+ 311us) server_negotiation.cc:407] Connection header received
0929 02:55:26.686650 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:26.686653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:26.686717 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:26.686801 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:26.688152 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.688877 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.690429 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.690836 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.690943 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:26.694573 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:26.694578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:26.694581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:26.694616 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:26.696978 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.697609 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.697613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.697616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.697748 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.698131 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.698134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.698136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.698473 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:26.698693 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:26.698982 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 02:55:26.699187 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 02:55:26.700130 18513 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57818
I20260929 02:55:26.701476 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18574
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650526, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:26.715901 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:26.702441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.15.18.65:44511)
0929 02:55:26.703031 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:26.703038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:26.703063 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:55:26.703121 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:26.703127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:26.703319 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:26.703448 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:26.704616 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.705609 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.706715 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.707279 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.707382 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:26.711031 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:26.711043 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:26.711058 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:26.711098 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:26.713692 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.714244 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.714251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.714253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.714379 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.714642 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.714645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.714647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.714993 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:26.715207 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:26.715382 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 02:55:26.715615 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":290,"threads_started":1}
W20260929 02:55:26.717074 18697 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55856
W20260929 02:55:26.718153 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18574: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:55:26.718338 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 18574
I20260929 02:55:26.730430 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18708
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650526, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:55:26.746021 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:26.731241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.15.18.66:38089)
0929 02:55:26.731744 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:26.731752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:26.731789 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:55:26.731849 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:26.731855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:26.732073 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:26.732193 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:26.733325 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.734291 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.735491 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.736163 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.736275 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:26.740100 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:26.740113 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:26.740128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:26.740163 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:26.743059 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.743846 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.743853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.743856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.743989 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.744428 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.744433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.744435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.744835 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:26.745092 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:26.745373 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 02:55:26.745617 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":266,"threads_started":1}
W20260929 02:55:26.747298 18831 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57298
W20260929 02:55:26.748515 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18708: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:55:26.748690 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 18708
I20260929 02:55:26.760852 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18841
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650526, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:55:26.778970 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:26.761751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.15.18.67:41881)
0929 02:55:26.762451 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:26.762457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:26.762487 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:55:26.762554 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:26.762560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:26.762771 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:26.762966 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:26.764542 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.765753 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.767064 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:26.767703 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:26.767824 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:26.772769 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:26.772784 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:26.772800 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:26.772838 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:26.775874 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.776665 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.776674 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.776677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.776824 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:26.777245 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:26.777250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:26.777254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:26.777624 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:26.777938 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:26.778237 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 02:55:26.778515 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":371,"threads_started":1}
W20260929 02:55:26.780265 18964 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41270
W20260929 02:55:26.781358 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18841: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:55:26.781533 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 18841
I20260929 02:55:26.793639 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18481
I20260929 02:55:26.794827 18543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:26.899041 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 18481
2026-09-29T02:55:26Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3483 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18986](info): set up 2 sockets
Sep 29 02:55:26 dist-test-slave-q5f0 krb5kdc[18986](info): commencing operation
krb5kdc: starting...
W20260929 02:55:27.900004 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:55:28 dist-test-slave-q5f0 krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:55:28.088308 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43433
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:28.407981 18999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:28.408275 18999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:28.408365 18999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:28.415791 18999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:28.415908 18999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:28.415964 18999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:28.416021 18999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:28.416064 18999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:28.426079 18999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:28.427403 18999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:28.428720 18999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:28.440511 19004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:28.440574 19007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:28.440591 19005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:28.441401 18999 server_base.cc:1034] running on GCE node
I20260929 02:55:28.442227 18999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:28.443703 18999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:28.444922 18999 hybrid_clock.cc:648] HybridClock initialized: now 1790650528444870 us; error 34 us; skew 500 ppm
I20260929 02:55:28.445264 18999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:28 dist-test-slave-q5f0 krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:28.448311 18999 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:55:28.449775 18999 webserver.cc:460] Webserver started at http://127.15.18.126:43323/ using document root <none> and password file <none>
I20260929 02:55:28.450446 18999 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:28.450555 18999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:28.450824 18999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:28.453737 18999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "785f669ab83e401ebb450b417b687671"
format_stamp: "Formatted at 2026-09-29 02:55:28 on dist-test-slave-q5f0"
server_key: "d0cfa1ca8e7b5907f2af6d5a5a6401da"
server_key_iv: "64773f9807e84edd05540d831966f649"
server_key_version: "encryptionkey@0"
I20260929 02:55:28.454483 18999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "785f669ab83e401ebb450b417b687671"
format_stamp: "Formatted at 2026-09-29 02:55:28 on dist-test-slave-q5f0"
server_key: "d0cfa1ca8e7b5907f2af6d5a5a6401da"
server_key_iv: "64773f9807e84edd05540d831966f649"
server_key_version: "encryptionkey@0"
I20260929 02:55:28.459693 18999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 02:55:28.463195 19014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:28.464627 18999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:28.464792 18999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "785f669ab83e401ebb450b417b687671"
format_stamp: "Formatted at 2026-09-29 02:55:28 on dist-test-slave-q5f0"
server_key: "d0cfa1ca8e7b5907f2af6d5a5a6401da"
server_key_iv: "64773f9807e84edd05540d831966f649"
server_key_version: "encryptionkey@0"
I20260929 02:55:28.464973 18999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:28.485812 18999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:28.490239 18999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:28.490512 18999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:28.507746 18999 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:43433
I20260929 02:55:28.507768 19065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:43433 every 8 connection(s)
I20260929 02:55:28.509071 18999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:28.512874 19066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:28.516222 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 18999
I20260929 02:55:28.516418 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:28.517055 15433 external_mini_cluster.cc:1428] Setting key fae58be0a451732dd8854770704e2bf0
I20260929 02:55:28.524896 19066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671: Bootstrap starting.
I20260929 02:55:28.528007 19066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:28.528999 19066 log.cc:824] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:55:28 dist-test-slave-q5f0 krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:28.532197 19066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671: No bootstrap required, opened a new log
I20260929 02:55:28.539808 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:28.519040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55830 (local address 127.15.18.126:43433)
0929 02:55:28.519849 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:28.519862 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:28.519904 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:55:28.521453 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:28.521476 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:28.521907 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:28.522317 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:28.523747 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:28.525083 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:28.526354 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:28.527020 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:28.527187 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:28.531190 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:28.531201 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:28.531221 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:28.531265 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:28.535173 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:28.535876 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:28.535886 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:28.535895 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:28.536103 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:28.536470 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:28.536475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:28.536479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:28.537145 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:28.537456 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:28.537876 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 02:55:28.538339 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":370,"threads_started":1}
I20260929 02:55:28.542500 19066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785f669ab83e401ebb450b417b687671" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43433 } }
I20260929 02:55:28.542946 19066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:28.543040 19066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785f669ab83e401ebb450b417b687671, State: Initialized, Role: FOLLOWER
I20260929 02:55:28.543603 19066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785f669ab83e401ebb450b417b687671" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43433 } }
I20260929 02:55:28.543825 19066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:28.543946 19066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:28.544093 19066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:28.546334 19066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785f669ab83e401ebb450b417b687671" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43433 } }
I20260929 02:55:28.546813 19066 leader_election.cc:302] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 785f669ab83e401ebb450b417b687671; no voters:
I20260929 02:55:28.547389 19066 leader_election.cc:288] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:28.547561 19071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:28.548317 19071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [term 1 LEADER]: Becoming Leader. State: Replica: 785f669ab83e401ebb450b417b687671, State: Running, Role: LEADER
I20260929 02:55:28.548892 19071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785f669ab83e401ebb450b417b687671" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43433 } }
I20260929 02:55:28.549625 19066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:28.551898 19072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "785f669ab83e401ebb450b417b687671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785f669ab83e401ebb450b417b687671" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43433 } } }
I20260929 02:55:28.552301 19072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:28.555796 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 785f669ab83e401ebb450b417b687671. Latest consensus state: current_term: 1 leader_uuid: "785f669ab83e401ebb450b417b687671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785f669ab83e401ebb450b417b687671" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43433 } } }
I20260929 02:55:28.556133 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:28.557756 19079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:28.563035 19079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:28.572145 19079 catalog_manager.cc:1347] Generated new cluster ID: f344d76f5bfb49baa1db312ffce51e9e
I20260929 02:55:28.572257 19079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:28.610569 19079 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:28.612002 19079 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:28.623001 19079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 785f669ab83e401ebb450b417b687671: Generated new TSK 0
I20260929 02:55:28.623910 19079 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:55:28.674911 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43433
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:28.997785 19092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:28.998091 19092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:28.998201 19092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:29.005569 19092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:29.005689 19092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:29.005843 19092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:29.016098 19092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:29.017422 19092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:29.018726 19092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:29.031700 19098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:29.031646 19097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:29.032117 19100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:29.032620 19092 server_base.cc:1034] running on GCE node
I20260929 02:55:29.033241 19092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:29.034219 19092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:29.035447 19092 hybrid_clock.cc:648] HybridClock initialized: now 1790650529035392 us; error 39 us; skew 500 ppm
I20260929 02:55:29.035794 19092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:29 dist-test-slave-q5f0 krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650529, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:29.039045 19092 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:55:29.040565 19092 webserver.cc:460] Webserver started at http://127.15.18.65:38213/ using document root <none> and password file <none>
I20260929 02:55:29.041263 19092 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:29.041369 19092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:29.041664 19092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:29.044565 19092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "2c8faca320ac47cc98c2c397f4a4dda6"
format_stamp: "Formatted at 2026-09-29 02:55:29 on dist-test-slave-q5f0"
server_key: "e1f606ab2c204107acbd81ba9f704ad9"
server_key_iv: "46fbfffc89282573ae0bc8fa8c51e54c"
server_key_version: "encryptionkey@0"
I20260929 02:55:29.045290 19092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "2c8faca320ac47cc98c2c397f4a4dda6"
format_stamp: "Formatted at 2026-09-29 02:55:29 on dist-test-slave-q5f0"
server_key: "e1f606ab2c204107acbd81ba9f704ad9"
server_key_iv: "46fbfffc89282573ae0bc8fa8c51e54c"
server_key_version: "encryptionkey@0"
I20260929 02:55:29.050732 19092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:55:29.054250 19107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:29.055689 19092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:29.055835 19092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "2c8faca320ac47cc98c2c397f4a4dda6"
format_stamp: "Formatted at 2026-09-29 02:55:29 on dist-test-slave-q5f0"
server_key: "e1f606ab2c204107acbd81ba9f704ad9"
server_key_iv: "46fbfffc89282573ae0bc8fa8c51e54c"
server_key_version: "encryptionkey@0"
I20260929 02:55:29.056028 19092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:29.073190 19092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:29.077421 19092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:29.077687 19092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:29.078655 19092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:29.080044 19092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:29.080118 19092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:29.080237 19092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:29.080287 19092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:29.110472 19092 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:46261
I20260929 02:55:29.110508 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:46261 every 8 connection(s)
I20260929 02:55:29.111801 19092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:29.112774 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 19092
I20260929 02:55:29.112984 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:29.113808 15433 external_mini_cluster.cc:1428] Setting key cbdc2c81060a6b2d8697ab90b55a60f3
Sep 29 02:55:29 dist-test-slave-q5f0 krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650529, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
I20260929 02:55:29.133922 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:29.114737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:60359 (local address 127.15.18.126:43433)
0929 02:55:29.115217 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:29.115224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:29.116345 (+ 1121us) server_negotiation.cc:407] Connection header received
0929 02:55:29.118185 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:29.118191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:29.118289 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:29.118399 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:29.120117 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:29.120863 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:29.123124 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:29.123701 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:29.123811 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:29.127982 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:29.127988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:29.127992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:29.128026 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:29.130638 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:29.131630 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:29.131635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:29.131637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:29.131759 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:29.132190 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:29.132194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:29.132197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:29.132527 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:29.132644 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:29.133405 (+ 761us) server_negotiation.cc:299] Negotiation successful
0929 02:55:29.133585 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":235,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
I20260929 02:55:29.134976 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:29.115343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43433 (local address 127.15.18.65:60359)
0929 02:55:29.116178 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:29.116226 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:29.117904 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:29.118664 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:29.118674 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:29.119261 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:29.119937 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:29.119957 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:29.121019 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:29.121024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:29.122957 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:29.122972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:29.123936 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:29.123943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:29.124224 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:29.124940 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:29.124976 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:29.127670 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:29.130808 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:29.130819 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:29.130839 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:29.131430 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:29.131859 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:29.131863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:29.131865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:29.132052 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:29.132663 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:29.132670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:29.133022 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 02:55:29.133261 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 02:55:29.133701 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":336,"threads_started":1}
I20260929 02:55:29.139629 19220 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43433
I20260929 02:55:29.139943 19220 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:29.140668 19220 heartbeater.cc:511] Master 127.15.18.126:43433 requested a full tablet report, sending...
I20260929 02:55:29.142962 19031 ts_manager.cc:194] Registered new tserver with Master: 2c8faca320ac47cc98c2c397f4a4dda6 (127.15.18.65:46261)
I20260929 02:55:29.144981 19031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:60359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:55:29.159776 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43433
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:29.485832 19226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:29.486189 19226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:29.486379 19226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:29.493723 19226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:29.493840 19226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:29.494010 19226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:29.503829 19226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:29.505167 19226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:29.506513 19226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:29.520473 19231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:29.520464 19232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:29.521225 19234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:29.521690 19226 server_base.cc:1034] running on GCE node
I20260929 02:55:29.522317 19226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:29.523280 19226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:29.524515 19226 hybrid_clock.cc:648] HybridClock initialized: now 1790650529524441 us; error 64 us; skew 500 ppm
I20260929 02:55:29.524904 19226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:29 dist-test-slave-q5f0 krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650529, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:29.528009 19226 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:55:29.529470 19226 webserver.cc:460] Webserver started at http://127.15.18.66:33251/ using document root <none> and password file <none>
I20260929 02:55:29.530130 19226 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:29.530228 19226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:29.530485 19226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:29.533354 19226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "9726cad2291d434d89326b399c52f619"
format_stamp: "Formatted at 2026-09-29 02:55:29 on dist-test-slave-q5f0"
server_key: "d93b79812908f339dea50a5b386149e1"
server_key_iv: "47afec03f2ad53874f781a00f262759a"
server_key_version: "encryptionkey@0"
I20260929 02:55:29.534068 19226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "9726cad2291d434d89326b399c52f619"
format_stamp: "Formatted at 2026-09-29 02:55:29 on dist-test-slave-q5f0"
server_key: "d93b79812908f339dea50a5b386149e1"
server_key_iv: "47afec03f2ad53874f781a00f262759a"
server_key_version: "encryptionkey@0"
I20260929 02:55:29.539151 19226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:29.542721 19241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:29.544256 19226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:29.544416 19226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "9726cad2291d434d89326b399c52f619"
format_stamp: "Formatted at 2026-09-29 02:55:29 on dist-test-slave-q5f0"
server_key: "d93b79812908f339dea50a5b386149e1"
server_key_iv: "47afec03f2ad53874f781a00f262759a"
server_key_version: "encryptionkey@0"
I20260929 02:55:29.544615 19226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:29.562695 19226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:29.566875 19226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:29.567157 19226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:29.568173 19226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:29.569541 19226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:29.569612 19226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:29.569731 19226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:29.569779 19226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:29.600118 19226 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:44413
I20260929 02:55:29.600159 19353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:44413 every 8 connection(s)
I20260929 02:55:29.601490 19226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:29.608709 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 19226
I20260929 02:55:29.608917 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:29.609448 15433 external_mini_cluster.cc:1428] Setting key f31153ab0322d913f48f2071124b63cb
Sep 29 02:55:29 dist-test-slave-q5f0 krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650529, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:29.623587 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:29.604236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:47953 (local address 127.15.18.126:43433)
0929 02:55:29.604402 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:29.604407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:29.605708 (+ 1301us) server_negotiation.cc:407] Connection header received
0929 02:55:29.607622 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:29.607627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:29.607704 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:29.607793 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:29.609933 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:29.610774 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:29.612988 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:29.613399 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:29.613520 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:29.617184 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:29.617191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:29.617197 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:29.617239 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:29.620357 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:29.621121 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:29.621127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:29.621130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:29.621289 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:29.621688 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:29.621692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:29.621695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:29.622014 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:29.622178 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:29.622888 (+ 710us) server_negotiation.cc:299] Negotiation successful
0929 02:55:29.623157 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:55:29.624567 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:29.604808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43433 (local address 127.15.18.66:47953)
0929 02:55:29.605530 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:29.605568 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:29.607288 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:29.608079 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:29.608088 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:29.608843 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:29.609712 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:29.609739 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:29.610969 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:29.610975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:29.612827 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:29.612844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:29.613639 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:29.613646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:29.613883 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:29.614550 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:29.614577 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:29.616918 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:29.620531 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:29.620539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:29.620556 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:29.621006 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:29.621399 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:29.621402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:29.621405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:29.621594 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:29.622170 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:29.622176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:29.622535 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 02:55:29.622781 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 02:55:29.623326 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":270,"threads_started":1}
I20260929 02:55:29.626053 19354 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43433
I20260929 02:55:29.626401 19354 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:29.627318 19354 heartbeater.cc:511] Master 127.15.18.126:43433 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
I20260929 02:55:29.629025 19031 ts_manager.cc:194] Registered new tserver with Master: 9726cad2291d434d89326b399c52f619 (127.15.18.66:44413)
I20260929 02:55:29.629913 19031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:47953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:55:29.653045 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43433
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:29.965790 19359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:29.966086 19359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:29.966203 19359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:29.973621 19359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:29.973742 19359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:29.973889 19359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:29.983729 19359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:29.985035 19359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:29.986374 19359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:29.999590 19365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:29.999626 19364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:29.999648 19367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:30.000272 19359 server_base.cc:1034] running on GCE node
I20260929 02:55:30.000864 19359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:30.001854 19359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:30.003088 19359 hybrid_clock.cc:648] HybridClock initialized: now 1790650530003021 us; error 56 us; skew 500 ppm
I20260929 02:55:30.003434 19359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:30 dist-test-slave-q5f0 krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650530, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:30.011279 19359 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:55:30.012936 19359 webserver.cc:460] Webserver started at http://127.15.18.67:44037/ using document root <none> and password file <none>
I20260929 02:55:30.013685 19359 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:30.013793 19359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:30.014104 19359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:30.017159 19359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "4bb228892ccc43928e8e16eb6296b4cf"
format_stamp: "Formatted at 2026-09-29 02:55:30 on dist-test-slave-q5f0"
server_key: "b2a62d1ae7b96e2428101c99a39947ef"
server_key_iv: "902144ccc50248151122050c90764313"
server_key_version: "encryptionkey@0"
I20260929 02:55:30.017967 19359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "4bb228892ccc43928e8e16eb6296b4cf"
format_stamp: "Formatted at 2026-09-29 02:55:30 on dist-test-slave-q5f0"
server_key: "b2a62d1ae7b96e2428101c99a39947ef"
server_key_iv: "902144ccc50248151122050c90764313"
server_key_version: "encryptionkey@0"
I20260929 02:55:30.023268 19359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260929 02:55:30.026947 19374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:30.028528 19359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:30.028676 19359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "4bb228892ccc43928e8e16eb6296b4cf"
format_stamp: "Formatted at 2026-09-29 02:55:30 on dist-test-slave-q5f0"
server_key: "b2a62d1ae7b96e2428101c99a39947ef"
server_key_iv: "902144ccc50248151122050c90764313"
server_key_version: "encryptionkey@0"
I20260929 02:55:30.028855 19359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:30.046120 19359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:30.050472 19359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:30.050729 19359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:30.051699 19359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:30.052994 19359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:30.053068 19359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:30.053184 19359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:30.053233 19359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:30.082521 19359 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:32869
I20260929 02:55:30.082564 19486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:32869 every 8 connection(s)
I20260929 02:55:30.083874 19359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:30.090855 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 19359
I20260929 02:55:30.091053 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:30.091606 15433 external_mini_cluster.cc:1428] Setting key 988c0730cd93440e023a36b389b36dc5
Sep 29 02:55:30 dist-test-slave-q5f0 krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650530, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:30.103751 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:30.086489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:38563 (local address 127.15.18.126:43433)
0929 02:55:30.086657 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:30.086661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:30.087853 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 02:55:30.089655 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:30.089660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:30.089742 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:30.089828 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:30.091730 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.092544 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.094451 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.094842 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.094943 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:30.098238 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:30.098244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:30.098248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:30.098281 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:30.100658 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.101364 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.101367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.101370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.101494 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.101981 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.101986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.101989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.102317 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:30.102459 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:30.103217 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 02:55:30.103425 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:55:30.104638 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:30.086977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43433 (local address 127.15.18.67:38563)
0929 02:55:30.087711 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:30.087749 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:30.089372 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:30.090088 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:30.090096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:30.090699 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:30.091533 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:30.091559 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.092711 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.092717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:30.094310 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:30.094324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.095067 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.095073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:30.095272 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:30.095868 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:30.095894 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:30.098041 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:30.100796 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:30.100803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:30.100817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:30.101264 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:30.101610 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:30.101614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:30.101616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:30.101827 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:30.102463 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:30.102470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:30.102810 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 02:55:30.103065 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 02:55:30.103550 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
I20260929 02:55:30.106132 19487 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43433
I20260929 02:55:30.106479 19487 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:30.107347 19487 heartbeater.cc:511] Master 127.15.18.126:43433 requested a full tablet report, sending...
I20260929 02:55:30.108837 19030 ts_manager.cc:194] Registered new tserver with Master: 4bb228892ccc43928e8e16eb6296b4cf (127.15.18.67:32869)
I20260929 02:55:30.109706 19030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:38563
I20260929 02:55:30.119532 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:55:30 dist-test-slave-q5f0 krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260929 02:55:30.148372 19220 heartbeater.cc:503] Master 127.15.18.126:43433 was elected leader, sending a full tablet report...
Sep 29 02:55:30 dist-test-slave-q5f0 krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:30.165304 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:30.150050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.15.18.65:46261)
0929 02:55:30.150897 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:30.150911 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:30.150934 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:55:30.150982 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:30.150988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:30.151234 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:30.151374 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:30.152751 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.153731 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.154959 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.155568 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.155684 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:30.159502 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:30.159536 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:30.159556 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:30.159601 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:30.162627 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.163337 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.163345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.163347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.163490 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.163928 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.163931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.163934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.164277 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:30.164533 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:30.164803 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 02:55:30.165019 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":528,"threads_started":1}
Sep 29 02:55:30 dist-test-slave-q5f0 krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:30.182547 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:30.169468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.15.18.126:43433)
0929 02:55:30.169669 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:30.169675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:30.169898 (+ 223us) server_negotiation.cc:407] Connection header received
0929 02:55:30.170088 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:30.170093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:30.170160 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:30.170240 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:30.171426 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.172212 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.173422 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.173815 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.173922 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:30.177672 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:30.177677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:30.177681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:30.177718 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:30.180343 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.180927 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.180932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.180934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.181064 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.181347 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.181350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.181352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.181677 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:30.181854 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:30.182157 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 02:55:30.182313 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 02:55:30.183136 19030 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55844
I20260929 02:55:30.184068 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 19092
I20260929 02:55:30.196003 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:30.185019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38278 (local address 127.15.18.65:46261)
0929 02:55:30.185220 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:30.185224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:30.185303 (+ 79us) server_negotiation.cc:407] Connection header received
0929 02:55:30.185448 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:30.185452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:30.185511 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:30.185586 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:30.186692 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.187360 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.188591 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.189023 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.189123 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:30.191303 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:30.191308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:30.191311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:30.191344 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:30.193799 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.194405 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.194408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.194410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.194522 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.194786 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.194789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.194791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.195046 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:30.195217 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:30.195440 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 02:55:30.195644 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:55:30.196720 19215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:30.278196 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 19092
I20260929 02:55:30.290930 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 19226
Sep 29 02:55:30 dist-test-slave-q5f0 krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:55:30.307449 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:30.291781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.15.18.66:44413)
0929 02:55:30.292318 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:30.292325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:30.292351 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:55:30.292415 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:30.292422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:30.292632 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:30.292792 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:30.294227 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.295268 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.296690 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.297272 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.297393 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:30.301590 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:30.301604 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:30.301620 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:30.301667 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:30.304430 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.305159 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.305167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.305170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.305306 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.305688 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.305692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.305694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.306050 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:30.306396 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:30.306843 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 02:55:30.307113 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":275,"threads_started":1}
I20260929 02:55:30.309820 19349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:30.415246 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 19226
I20260929 02:55:30.427413 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 19359
Sep 29 02:55:30 dist-test-slave-q5f0 krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:55:30.442937 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:30.428382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.15.18.67:32869)
0929 02:55:30.429006 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:30.429014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:30.429043 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:55:30.429107 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:30.429113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:30.429332 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:30.429463 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:30.430693 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.431766 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.433224 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:30.433763 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:30.433858 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:30.437941 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:30.437953 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:30.437968 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:30.438001 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:30.440614 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.441201 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.441207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.441210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.441326 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:30.441595 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:30.441597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:30.441600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:30.441926 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:30.442145 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:30.442360 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 02:55:30.442558 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":290,"threads_started":1}
I20260929 02:55:30.444228 19482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:30.526214 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 19359
I20260929 02:55:30.538770 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 18999
I20260929 02:55:30.539971 19061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:30.641620 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 18999
2026-09-29T02:55:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3742 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:30 dist-test-slave-q5f0 krb5kdc[19506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:30 dist-test-slave-q5f0 krb5kdc[19506](info): set up 2 sockets
Sep 29 02:55:30 dist-test-slave-q5f0 krb5kdc[19506](info): commencing operation
krb5kdc: starting...
W20260929 02:55:31.644150 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:55:31 dist-test-slave-q5f0 krb5kdc[19506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:55:31.834882 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43471
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:32.147811 19519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:32.148140 19519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:32.148281 19519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:32.155534 19519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:32.155653 19519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:32.155725 19519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:32.155795 19519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:32.155858 19519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:32.166035 19519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:32.167371 19519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:32.168720 19519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:32.180035 19525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:32.180351 19527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:32.179915 19524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:32.180691 19519 server_base.cc:1034] running on GCE node
I20260929 02:55:32.181555 19519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:32.183070 19519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:32.184276 19519 hybrid_clock.cc:648] HybridClock initialized: now 1790650532184230 us; error 39 us; skew 500 ppm
I20260929 02:55:32.184646 19519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:32 dist-test-slave-q5f0 krb5kdc[19506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650532, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:32.187871 19519 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:55:32.189025 19519 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:32.189127 19519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:32.189447 19519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:32.192581 19519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "1d5832102cfb44bc9de5215046e78f3f"
format_stamp: "Formatted at 2026-09-29 02:55:32 on dist-test-slave-q5f0"
server_key: "9533723356abf666ed847639283915a2"
server_key_iv: "c6fb409210c5e04845fcf93a6346706d"
server_key_version: "encryptionkey@0"
I20260929 02:55:32.193382 19519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "1d5832102cfb44bc9de5215046e78f3f"
format_stamp: "Formatted at 2026-09-29 02:55:32 on dist-test-slave-q5f0"
server_key: "9533723356abf666ed847639283915a2"
server_key_iv: "c6fb409210c5e04845fcf93a6346706d"
server_key_version: "encryptionkey@0"
I20260929 02:55:32.198719 19519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:32.202500 19533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:32.204089 19519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:32.204279 19519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "1d5832102cfb44bc9de5215046e78f3f"
format_stamp: "Formatted at 2026-09-29 02:55:32 on dist-test-slave-q5f0"
server_key: "9533723356abf666ed847639283915a2"
server_key_iv: "c6fb409210c5e04845fcf93a6346706d"
server_key_version: "encryptionkey@0"
I20260929 02:55:32.204489 19519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:32.218348 19519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:32.222616 19519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:32.222882 19519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:32.240919 19519 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:43471
I20260929 02:55:32.240900 19584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:43471 every 8 connection(s)
I20260929 02:55:32.241840 19519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:32.242895 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 19519
I20260929 02:55:32.243113 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:32.243938 15433 external_mini_cluster.cc:1428] Setting key bf1958197c81dc4cc7ae5c1302133f88
I20260929 02:55:32.246671 19585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:55:32 dist-test-slave-q5f0 krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:32.259862 19585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f: Bootstrap starting.
I20260929 02:55:32.263542 19585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:32.264849 19585 log.cc:824] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:32.268046 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:32.246042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.15.18.126:43471)
0929 02:55:32.246796 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:32.246809 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:32.246850 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:55:32.248505 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:32.248527 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:32.248940 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:32.249301 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:32.251143 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:32.252600 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:32.254346 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:32.255066 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:32.255199 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:32.260062 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:32.260075 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:32.260092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:32.260140 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:32.263172 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:32.264107 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:32.264114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:32.264121 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:32.264288 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:32.264703 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:32.264707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:32.264709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:32.265323 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:32.265646 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:32.266143 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 02:55:32.266624 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":331,"threads_started":1}
I20260929 02:55:32.268465 19585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f: No bootstrap required, opened a new log
I20260929 02:55:32.276252 19585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5832102cfb44bc9de5215046e78f3f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43471 } }
I20260929 02:55:32.276615 19585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:32.276695 19585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5832102cfb44bc9de5215046e78f3f, State: Initialized, Role: FOLLOWER
I20260929 02:55:32.277235 19585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5832102cfb44bc9de5215046e78f3f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43471 } }
I20260929 02:55:32.277454 19585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:32.277549 19585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:32.277676 19585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:32.279790 19585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5832102cfb44bc9de5215046e78f3f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43471 } }
I20260929 02:55:32.280236 19585 leader_election.cc:302] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d5832102cfb44bc9de5215046e78f3f; no voters:
I20260929 02:55:32.280761 19585 leader_election.cc:288] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:32.280961 19590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:32.281618 19590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [term 1 LEADER]: Becoming Leader. State: Replica: 1d5832102cfb44bc9de5215046e78f3f, State: Running, Role: LEADER
I20260929 02:55:32.282130 19590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5832102cfb44bc9de5215046e78f3f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43471 } }
I20260929 02:55:32.282756 19585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:32.285079 19591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d5832102cfb44bc9de5215046e78f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5832102cfb44bc9de5215046e78f3f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43471 } } }
I20260929 02:55:32.285516 19591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:32.285101 19592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d5832102cfb44bc9de5215046e78f3f. Latest consensus state: current_term: 1 leader_uuid: "1d5832102cfb44bc9de5215046e78f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5832102cfb44bc9de5215046e78f3f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43471 } } }
I20260929 02:55:32.286293 19592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:32.288662 19599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:32.292956 19599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:32.302132 19599 catalog_manager.cc:1347] Generated new cluster ID: 11e712f4d5ba4ad7831ee358cf893b79
I20260929 02:55:32.302245 19599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:32.322377 19599 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:32.324164 19599 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:32.333873 19599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f: Generated new TSK 0
I20260929 02:55:32.334748 19599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:55:32.379017 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43471
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:32.695870 19611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:32.696161 19611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:32.696287 19611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:32.703552 19611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:32.703665 19611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:32.703819 19611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:32.713761 19611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:32.715147 19611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:32.716581 19611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:32.729269 19616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:32.729238 19619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:32.729303 19617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:32.729875 19611 server_base.cc:1034] running on GCE node
I20260929 02:55:32.730512 19611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:32.731436 19611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:32.732678 19611 hybrid_clock.cc:648] HybridClock initialized: now 1790650532732613 us; error 52 us; skew 500 ppm
I20260929 02:55:32.733009 19611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:32 dist-test-slave-q5f0 krb5kdc[19506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650532, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:32.735947 19611 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:55:32.737030 19611 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:32.737134 19611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:32.737484 19611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:32.740403 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "30411bca67454363a8e3e4682d41e58e"
format_stamp: "Formatted at 2026-09-29 02:55:32 on dist-test-slave-q5f0"
server_key: "7965efa44f1c698dc2b035137748120b"
server_key_iv: "4bea5991f297def69b828e10330fb324"
server_key_version: "encryptionkey@0"
I20260929 02:55:32.741163 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "30411bca67454363a8e3e4682d41e58e"
format_stamp: "Formatted at 2026-09-29 02:55:32 on dist-test-slave-q5f0"
server_key: "7965efa44f1c698dc2b035137748120b"
server_key_iv: "4bea5991f297def69b828e10330fb324"
server_key_version: "encryptionkey@0"
I20260929 02:55:32.746091 19611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 02:55:32.749572 19625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:32.750954 19611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:55:32.751120 19611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "30411bca67454363a8e3e4682d41e58e"
format_stamp: "Formatted at 2026-09-29 02:55:32 on dist-test-slave-q5f0"
server_key: "7965efa44f1c698dc2b035137748120b"
server_key_iv: "4bea5991f297def69b828e10330fb324"
server_key_version: "encryptionkey@0"
I20260929 02:55:32.751322 19611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:32.764250 19611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:32.768728 19611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:32.768980 19611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:32.769937 19611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:32.771245 19611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:32.771337 19611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:32.771438 19611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:32.771523 19611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:32.799048 19611 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:42513
I20260929 02:55:32.799095 19737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:42513 every 8 connection(s)
I20260929 02:55:32.799913 19611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:32.809082 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 19611
I20260929 02:55:32.809278 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:32.810050 15433 external_mini_cluster.cc:1428] Setting key 534fc58e653643a7e89a1f395d623821
Sep 29 02:55:32 dist-test-slave-q5f0 krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650532, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:32.819826 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:32.802298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:42951 (local address 127.15.18.126:43471)
0929 02:55:32.802729 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:32.802734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:32.803547 (+ 813us) server_negotiation.cc:407] Connection header received
0929 02:55:32.805281 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:32.805287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:32.805367 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:32.805457 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:32.806947 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:32.807708 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:32.809475 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:32.810093 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:32.810225 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:32.813916 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:32.813921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:32.813925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:32.813961 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:32.816580 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:32.817362 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:32.817368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:32.817373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:32.817499 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:32.817943 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:32.817949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:32.817953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:32.818265 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:32.818408 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:32.819148 (+ 740us) server_negotiation.cc:299] Negotiation successful
0929 02:55:32.819388 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":238,"threads_started":1}
I20260929 02:55:32.820827 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:32.802710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43471 (local address 127.15.18.65:42951)
0929 02:55:32.803370 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:32.803416 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:32.805018 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:32.805690 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:32.805698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:32.806203 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:32.806780 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:32.806797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:32.807843 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:32.807847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:32.809285 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:32.809297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:32.810311 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:32.810316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:32.810510 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:32.811141 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:32.811168 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:32.813630 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:32.816745 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:32.816756 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:32.816778 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:32.817234 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:32.817611 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:32.817617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:32.817621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:32.817831 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:32.818396 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:32.818405 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:32.818774 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 02:55:32.819049 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 02:55:32.819624 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":255,"threads_started":1}
I20260929 02:55:32.822358 19738 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43471
I20260929 02:55:32.822649 19738 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:32.823364 19738 heartbeater.cc:511] Master 127.15.18.126:43471 requested a full tablet report, sending...
I20260929 02:55:32.825517 19550 ts_manager.cc:194] Registered new tserver with Master: 30411bca67454363a8e3e4682d41e58e (127.15.18.65:42513)
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
I20260929 02:55:32.827140 19550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:42951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:55:32.850965 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43471
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:33.162833 19744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:33.163121 19744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:33.163221 19744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:33.170477 19744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:33.170595 19744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:33.170711 19744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:33.180459 19744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:33.181726 19744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:33.183053 19744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:33.195726 19750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:33.195788 19749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:33.195827 19752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:33.196487 19744 server_base.cc:1034] running on GCE node
I20260929 02:55:33.197160 19744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:33.198128 19744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:33.199368 19744 hybrid_clock.cc:648] HybridClock initialized: now 1790650533199308 us; error 52 us; skew 500 ppm
I20260929 02:55:33.199735 19744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:33 dist-test-slave-q5f0 krb5kdc[19506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650533, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:33.203027 19744 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:55:33.204284 19744 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:33.204473 19744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:33.204813 19744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:33.208106 19744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "4704cff541e146169860dc7de666f6c2"
format_stamp: "Formatted at 2026-09-29 02:55:33 on dist-test-slave-q5f0"
server_key: "82d32b5648077586ff42a14d1755baaf"
server_key_iv: "4d5c01fc49c9a62f1377df8ed9ab6a25"
server_key_version: "encryptionkey@0"
I20260929 02:55:33.208946 19744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "4704cff541e146169860dc7de666f6c2"
format_stamp: "Formatted at 2026-09-29 02:55:33 on dist-test-slave-q5f0"
server_key: "82d32b5648077586ff42a14d1755baaf"
server_key_iv: "4d5c01fc49c9a62f1377df8ed9ab6a25"
server_key_version: "encryptionkey@0"
I20260929 02:55:33.214386 19744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:55:33.218214 19758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:33.219800 19744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:55:33.219990 19744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "4704cff541e146169860dc7de666f6c2"
format_stamp: "Formatted at 2026-09-29 02:55:33 on dist-test-slave-q5f0"
server_key: "82d32b5648077586ff42a14d1755baaf"
server_key_iv: "4d5c01fc49c9a62f1377df8ed9ab6a25"
server_key_version: "encryptionkey@0"
I20260929 02:55:33.220211 19744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:33.235129 19744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:33.239460 19744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:33.239763 19744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:33.240780 19744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:33.242178 19744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:33.242272 19744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:33.242372 19744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:33.242439 19744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:33.272284 19744 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:38257
I20260929 02:55:33.272358 19870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:38257 every 8 connection(s)
I20260929 02:55:33.273116 19744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:33.279254 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 19744
I20260929 02:55:33.279433 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:33.279870 15433 external_mini_cluster.cc:1428] Setting key a8f9017c622d5facd5688b673d7f9085
Sep 29 02:55:33 dist-test-slave-q5f0 krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650533, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:33.293803 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.275579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:37487 (local address 127.15.18.126:43471)
0929 02:55:33.275772 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:33.275778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:33.277020 (+ 1242us) server_negotiation.cc:407] Connection header received
0929 02:55:33.278800 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:33.278804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:33.278892 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:33.279013 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:33.280669 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.281717 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.283367 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.283846 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.283973 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.287499 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:33.287525 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:33.287530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:33.287573 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:33.290596 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:33.291487 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:33.291492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:33.291494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:33.291664 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:33.292107 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:33.292110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:33.292112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:33.292374 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:33.292474 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:33.293344 (+ 870us) server_negotiation.cc:299] Negotiation successful
0929 02:55:33.293528 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:55:33.295324 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.276135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43471 (local address 127.15.18.66:37487)
0929 02:55:33.276860 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:33.276898 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:33.278490 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:33.279195 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:33.279204 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:33.279870 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:33.280488 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:33.280508 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.281890 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.281896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:33.283220 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:33.283232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.284102 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.284110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:33.284334 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.284971 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:33.284998 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:33.287261 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:33.290775 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:33.290784 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:33.290802 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:33.291347 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:33.291763 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:33.291767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:33.291771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:33.292000 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:33.292497 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:33.292506 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:33.292922 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 02:55:33.293226 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 02:55:33.293786 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":287,"threads_started":1}
I20260929 02:55:33.296778 19871 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43471
I20260929 02:55:33.297087 19871 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:33.297717 19871 heartbeater.cc:511] Master 127.15.18.126:43471 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
I20260929 02:55:33.299154 19550 ts_manager.cc:194] Registered new tserver with Master: 4704cff541e146169860dc7de666f6c2 (127.15.18.66:38257)
I20260929 02:55:33.300004 19550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:37487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:55:33.323858 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43471
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:33.640484 19876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:33.640772 19876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:33.640869 19876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:33.648229 19876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:33.648350 19876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:33.648483 19876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:33.658334 19876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:33.659602 19876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:33.660877 19876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:33.673297 19881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:33.673292 19882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:33.673592 19884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:33.674093 19876 server_base.cc:1034] running on GCE node
I20260929 02:55:33.674665 19876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:33.675658 19876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:33.676883 19876 hybrid_clock.cc:648] HybridClock initialized: now 1790650533676817 us; error 55 us; skew 500 ppm
I20260929 02:55:33.677199 19876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:33 dist-test-slave-q5f0 krb5kdc[19506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650533, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:33.680482 19876 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:55:33.681576 19876 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:33.681664 19876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:33.681926 19876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:33.684989 19876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "c36f02192169449c9f23d1ffb7b4b7c1"
format_stamp: "Formatted at 2026-09-29 02:55:33 on dist-test-slave-q5f0"
server_key: "82f363017e2e931815cf4e21793aad71"
server_key_iv: "f7cf2a518c2f66dd94692421aba07f64"
server_key_version: "encryptionkey@0"
I20260929 02:55:33.685787 19876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "c36f02192169449c9f23d1ffb7b4b7c1"
format_stamp: "Formatted at 2026-09-29 02:55:33 on dist-test-slave-q5f0"
server_key: "82f363017e2e931815cf4e21793aad71"
server_key_iv: "f7cf2a518c2f66dd94692421aba07f64"
server_key_version: "encryptionkey@0"
I20260929 02:55:33.691141 19876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:33.694775 19890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:33.696185 19876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:33.696352 19876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "c36f02192169449c9f23d1ffb7b4b7c1"
format_stamp: "Formatted at 2026-09-29 02:55:33 on dist-test-slave-q5f0"
server_key: "82f363017e2e931815cf4e21793aad71"
server_key_iv: "f7cf2a518c2f66dd94692421aba07f64"
server_key_version: "encryptionkey@0"
I20260929 02:55:33.696563 19876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:33.710263 19876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:33.714550 19876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:33.714818 19876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:33.715919 19876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:33.717376 19876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:33.717489 19876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:33.717592 19876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:33.717648 19876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:33.746915 19876 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:35339
I20260929 02:55:33.746960 20002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:35339 every 8 connection(s)
I20260929 02:55:33.747844 19876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:33.751670 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 19876
I20260929 02:55:33.751863 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:33.752274 15433 external_mini_cluster.cc:1428] Setting key a8d9492b5404b9323fe5640b5310875b
Sep 29 02:55:33 dist-test-slave-q5f0 krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650533, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:33.767827 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.750234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:35285 (local address 127.15.18.126:43471)
0929 02:55:33.750389 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:33.750394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:33.751568 (+ 1174us) server_negotiation.cc:407] Connection header received
0929 02:55:33.753586 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:33.753591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:33.753676 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:33.753805 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:33.755727 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.756525 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.758036 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.758418 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.758505 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.761963 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:33.761968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:33.761972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:33.762008 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:33.764533 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:33.765338 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:33.765342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:33.765345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:33.765475 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:33.765913 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:33.765917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:33.765919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:33.766222 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:33.766376 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:33.767202 (+ 826us) server_negotiation.cc:299] Negotiation successful
0929 02:55:33.767429 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:55:33.768837 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.750706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43471 (local address 127.15.18.67:35285)
0929 02:55:33.751405 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:33.751444 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:33.753294 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:33.754034 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:33.754047 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:33.754771 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:33.755572 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:33.755592 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.756653 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.756656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:33.757896 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:33.757908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.758589 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.758593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:33.758782 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.759441 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:33.759468 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:33.761706 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:33.764675 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:33.764682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:33.764697 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:33.765173 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:33.765603 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:33.765607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:33.765610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:33.765790 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:33.766363 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:33.766372 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:33.766772 (+ 400us) client_negotiation.cc:769] Sending connection context
0929 02:55:33.767098 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 02:55:33.767701 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":267,"threads_started":1}
I20260929 02:55:33.770172 20003 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43471
I20260929 02:55:33.770470 20003 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:33.771135 20003 heartbeater.cc:511] Master 127.15.18.126:43471 requested a full tablet report, sending...
I20260929 02:55:33.772534 19550 ts_manager.cc:194] Registered new tserver with Master: c36f02192169449c9f23d1ffb7b4b7c1 (127.15.18.67:35339)
I20260929 02:55:33.773313 19550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:35285
I20260929 02:55:33.779875 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:55:33.797715 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.784240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.15.18.126:43471)
0929 02:55:33.784462 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:33.784469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:33.784823 (+ 354us) server_negotiation.cc:407] Connection header received
0929 02:55:33.784948 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:33.784952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:33.785013 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:33.785092 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:33.786416 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.787133 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.788479 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.788975 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.789065 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.791266 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:33.791271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:33.791275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:33.791309 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:33.794488 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:33.795279 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:33.795284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:33.795288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:33.795465 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:33.796065 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:33.796069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:33.796072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:33.796524 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:33.796759 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:33.797151 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 02:55:33.797373 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:55:33.802361 19550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:33.809007 19550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:33.830843 19738 heartbeater.cc:503] Master 127.15.18.126:43471 was elected leader, sending a full tablet report...
I20260929 02:55:33.843662 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.820947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.15.18.65:42513)
0929 02:55:33.834913 (+ 13966us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:33.834924 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:33.834958 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:55:33.835083 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:33.835092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:33.835386 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:33.835639 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:33.836647 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.838326 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.840943 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.842214 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.842337 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.842574 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:33.843123 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 02:55:33.843293 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13781,"thread_start_us":13630,"threads_started":1}
I20260929 02:55:33.843828 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.820745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:42513 (local address 127.0.0.1:39848)
0929 02:55:33.833995 (+ 13250us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:33.834046 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:33.834266 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:33.835967 (+ 1701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:33.835978 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:33.836016 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:33.836445 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:33.836460 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.838519 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.838524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:33.840771 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:33.840789 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.842580 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.842592 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:33.842812 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.842896 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 02:55:33.843060 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 02:55:33.843392 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12912,"mutex_wait_us":86,"thread_start_us":18773,"threads_started":1}
I20260929 02:55:33.847446 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.821750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.15.18.67:35339)
0929 02:55:33.838677 (+ 16927us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:33.838686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:33.839751 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 02:55:33.839842 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:33.839848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:33.840007 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:33.840177 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:33.841121 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.842586 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.844921 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.846064 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.846194 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.846420 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:33.846976 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 02:55:33.847130 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16741,"thread_start_us":16932,"threads_started":1}
I20260929 02:55:33.847582 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.820840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:38257 (local address 127.0.0.1:36996)
0929 02:55:33.837007 (+ 16167us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:33.837028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:33.837133 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:33.840132 (+ 2999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:33.840137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:33.840161 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:33.840612 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:33.840624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.842427 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.842433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:33.844525 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:33.844538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.846520 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.846527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:33.846709 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.846827 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 02:55:33.846948 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 02:55:33.847079 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16041,"mutex_wait_us":120,"thread_start_us":12656,"threads_started":1}
I20260929 02:55:33.847626 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.821190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:35339 (local address 127.0.0.1:33414)
0929 02:55:33.839496 (+ 18306us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:33.839530 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:33.839643 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:33.840447 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:33.840451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:33.840476 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:33.840930 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:33.840941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.842724 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.842729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:33.844769 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:33.844782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.846562 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.846569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:33.846778 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.846820 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:55:33.846901 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:55:33.847094 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18161,"mutex_wait_us":50,"thread_start_us":15563,"threads_started":1}
I20260929 02:55:33.848469 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:33.821516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36996 (local address 127.15.18.66:38257)
0929 02:55:33.839531 (+ 18015us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:33.839541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:33.839571 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:55:33.839661 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:33.839669 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:33.839930 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:33.840117 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:33.840784 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.842274 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.844684 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:33.846051 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:33.846182 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:33.846377 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:33.847985 (+ 1608us) server_negotiation.cc:299] Negotiation successful
0929 02:55:33.848148 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17573,"spinlock_wait_cycles":499456,"thread_start_us":17464,"threads_started":1}
I20260929 02:55:33.848295 19673 tablet_service.cc:1467] Processing CreateTablet for tablet 892fbb9f08ec4990abdb610bc7a72262 (DEFAULT_TABLE table=test-table [id=91b26eef135a4a8b91dea1a1c628387c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:33.849824 19673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892fbb9f08ec4990abdb610bc7a72262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:33.851269 19938 tablet_service.cc:1467] Processing CreateTablet for tablet 892fbb9f08ec4990abdb610bc7a72262 (DEFAULT_TABLE table=test-table [id=91b26eef135a4a8b91dea1a1c628387c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:33.851861 19806 tablet_service.cc:1467] Processing CreateTablet for tablet 892fbb9f08ec4990abdb610bc7a72262 (DEFAULT_TABLE table=test-table [id=91b26eef135a4a8b91dea1a1c628387c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:33.852664 19938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892fbb9f08ec4990abdb610bc7a72262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:33.853142 19806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892fbb9f08ec4990abdb610bc7a72262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:33.861069 20021 tablet_bootstrap.cc:492] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: Bootstrap starting.
I20260929 02:55:33.864266 20021 tablet_bootstrap.cc:654] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:33.865694 20021 log.cc:824] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:33.868856 20024 tablet_bootstrap.cc:492] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: Bootstrap starting.
I20260929 02:55:33.870474 20023 tablet_bootstrap.cc:492] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: Bootstrap starting.
I20260929 02:55:33.872967 20024 tablet_bootstrap.cc:654] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:33.873167 20023 tablet_bootstrap.cc:654] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:33.874357 20024 log.cc:824] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:33.875636 20023 log.cc:824] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:33.877086 20021 tablet_bootstrap.cc:492] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: No bootstrap required, opened a new log
I20260929 02:55:33.877338 20024 tablet_bootstrap.cc:492] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: No bootstrap required, opened a new log
I20260929 02:55:33.877341 20021 ts_tablet_manager.cc:1397] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20260929 02:55:33.877591 20024 ts_tablet_manager.cc:1397] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:55:33.878479 20023 tablet_bootstrap.cc:492] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: No bootstrap required, opened a new log
I20260929 02:55:33.878714 20023 ts_tablet_manager.cc:1397] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 02:55:33.884639 20021 raft_consensus.cc:348] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30411bca67454363a8e3e4682d41e58e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 42513 } } peers { permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 } } peers { permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 } }
I20260929 02:55:33.885308 20021 raft_consensus.cc:374] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:33.885452 20021 raft_consensus.cc:729] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30411bca67454363a8e3e4682d41e58e, State: Initialized, Role: FOLLOWER
I20260929 02:55:33.886036 20021 consensus_queue.cc:260] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30411bca67454363a8e3e4682d41e58e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 42513 } } peers { permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 } } peers { permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 } }
I20260929 02:55:33.887850 20021 ts_tablet_manager.cc:1428] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260929 02:55:33.888365 20024 raft_consensus.cc:348] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30411bca67454363a8e3e4682d41e58e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 42513 } } peers { permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 } } peers { permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 } }
I20260929 02:55:33.888880 20024 raft_consensus.cc:374] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:33.888698 20023 raft_consensus.cc:348] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30411bca67454363a8e3e4682d41e58e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 42513 } } peers { permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 } } peers { permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 } }
I20260929 02:55:33.889006 20024 raft_consensus.cc:729] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c36f02192169449c9f23d1ffb7b4b7c1, State: Initialized, Role: FOLLOWER
I20260929 02:55:33.889106 20023 raft_consensus.cc:374] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:33.889248 20023 raft_consensus.cc:729] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4704cff541e146169860dc7de666f6c2, State: Initialized, Role: FOLLOWER
I20260929 02:55:33.889792 20023 consensus_queue.cc:260] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30411bca67454363a8e3e4682d41e58e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 42513 } } peers { permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 } } peers { permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 } }
I20260929 02:55:33.889798 20024 consensus_queue.cc:260] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30411bca67454363a8e3e4682d41e58e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 42513 } } peers { permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 } } peers { permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 } }
I20260929 02:55:33.891324 20003 heartbeater.cc:503] Master 127.15.18.126:43471 was elected leader, sending a full tablet report...
I20260929 02:55:33.891635 20023 ts_tablet_manager.cc:1428] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260929 02:55:33.891927 20024 ts_tablet_manager.cc:1428] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 02:55:33.892158 19871 heartbeater.cc:503] Master 127.15.18.126:43471 was elected leader, sending a full tablet report...
I20260929 02:55:33.997677 20027 raft_consensus.cc:482] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:33.998016 20027 raft_consensus.cc:504] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30411bca67454363a8e3e4682d41e58e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 42513 } } peers { permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 } } peers { permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 } }
W20260929 02:55:33.999270 20004 tablet.cc:2369] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:33.999976 20027 leader_election.cc:288] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4704cff541e146169860dc7de666f6c2 (127.15.18.66:38257), c36f02192169449c9f23d1ffb7b4b7c1 (127.15.18.67:35339)
I20260929 02:55:34.006762 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:34.000547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:53095 (local address 127.15.18.67:35339)
0929 02:55:34.000718 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:34.000722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:34.001381 (+ 659us) server_negotiation.cc:407] Connection header received
0929 02:55:34.001618 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:34.001624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:34.001690 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:34.001757 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:34.002418 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.003144 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.005082 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.005898 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.006055 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:34.006126 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:34.006409 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 02:55:34.006531 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 02:55:34.006760 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:34.000363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:60063 (local address 127.15.18.66:38257)
0929 02:55:34.000595 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:34.000601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:34.001483 (+ 882us) server_negotiation.cc:407] Connection header received
0929 02:55:34.001637 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:34.001642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:34.001712 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:34.001796 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:34.002486 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.003172 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.005069 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.005837 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.005977 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:34.006054 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:34.006387 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 02:55:34.006520 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 02:55:34.006916 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:34.000389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:38257 (local address 127.15.18.65:60063)
0929 02:55:34.001377 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:34.001391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:34.001510 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:34.001823 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:34.001828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:34.001851 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:34.002304 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:34.002313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.003313 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.003316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:34.004910 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:34.004926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.006037 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.006043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:34.006222 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:34.006259 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:55:34.006332 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 02:55:34.006470 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":24,"thread_start_us":725,"threads_started":1}
I20260929 02:55:34.006958 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:34.000323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:35339 (local address 127.15.18.65:53095)
0929 02:55:34.001240 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:34.001269 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:34.001452 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:34.001797 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:34.001802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:34.001824 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:34.002277 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:34.002287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.003294 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.003299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:34.004936 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:34.004948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.006113 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.006117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:34.006263 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:34.006285 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:55:34.006369 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 02:55:34.006495 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":96,"thread_start_us":420,"threads_started":1}
I20260929 02:55:34.007884 19826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892fbb9f08ec4990abdb610bc7a72262" candidate_uuid: "30411bca67454363a8e3e4682d41e58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4704cff541e146169860dc7de666f6c2" is_pre_election: true
I20260929 02:55:34.007983 19958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892fbb9f08ec4990abdb610bc7a72262" candidate_uuid: "30411bca67454363a8e3e4682d41e58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" is_pre_election: true
I20260929 02:55:34.008396 19826 raft_consensus.cc:2454] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30411bca67454363a8e3e4682d41e58e in term 0.
I20260929 02:55:34.008416 19958 raft_consensus.cc:2454] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30411bca67454363a8e3e4682d41e58e in term 0.
I20260929 02:55:34.009006 19629 leader_election.cc:302] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30411bca67454363a8e3e4682d41e58e, 4704cff541e146169860dc7de666f6c2; no voters:
I20260929 02:55:34.009379 20027 raft_consensus.cc:2781] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:34.009536 20027 raft_consensus.cc:482] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:34.009621 20027 raft_consensus.cc:3037] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:34.011757 20027 raft_consensus.cc:504] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30411bca67454363a8e3e4682d41e58e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 42513 } } peers { permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 } } peers { permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 } }
I20260929 02:55:34.012328 20027 leader_election.cc:288] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [CANDIDATE]: Term 1 election: Requested vote from peers 4704cff541e146169860dc7de666f6c2 (127.15.18.66:38257), c36f02192169449c9f23d1ffb7b4b7c1 (127.15.18.67:35339)
I20260929 02:55:34.012835 19826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892fbb9f08ec4990abdb610bc7a72262" candidate_uuid: "30411bca67454363a8e3e4682d41e58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4704cff541e146169860dc7de666f6c2"
I20260929 02:55:34.012844 19958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892fbb9f08ec4990abdb610bc7a72262" candidate_uuid: "30411bca67454363a8e3e4682d41e58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c36f02192169449c9f23d1ffb7b4b7c1"
I20260929 02:55:34.013020 19826 raft_consensus.cc:3037] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:34.013029 19958 raft_consensus.cc:3037] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:34.015374 19826 raft_consensus.cc:2454] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30411bca67454363a8e3e4682d41e58e in term 1.
I20260929 02:55:34.015437 19958 raft_consensus.cc:2454] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30411bca67454363a8e3e4682d41e58e in term 1.
I20260929 02:55:34.015831 19629 leader_election.cc:302] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30411bca67454363a8e3e4682d41e58e, 4704cff541e146169860dc7de666f6c2; no voters:
I20260929 02:55:34.016124 20027 raft_consensus.cc:2781] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:34.016539 20027 raft_consensus.cc:686] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 1 LEADER]: Becoming Leader. State: Replica: 30411bca67454363a8e3e4682d41e58e, State: Running, Role: LEADER
I20260929 02:55:34.017012 20027 consensus_queue.cc:237] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30411bca67454363a8e3e4682d41e58e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 42513 } } peers { permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 } } peers { permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 } }
I20260929 02:55:34.021337 19549 catalog_manager.cc:5495] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e reported cstate change: term changed from 0 to 1, leader changed from <none> to 30411bca67454363a8e3e4682d41e58e (127.15.18.65). New cstate: current_term: 1 leader_uuid: "30411bca67454363a8e3e4682d41e58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30411bca67454363a8e3e4682d41e58e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 42513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 } health_report { overall_health: UNKNOWN } } }
W20260929 02:55:34.024870 19872 tablet.cc:2369] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:34.037025 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:34.031719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.15.18.65:42513)
0929 02:55:34.031880 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:34.031885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:34.032052 (+ 167us) server_negotiation.cc:407] Connection header received
0929 02:55:34.032235 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:34.032241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:34.032343 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:34.032521 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:34.032841 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.033622 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.034922 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.035302 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.035401 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:34.035937 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:34.036517 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:34.036644 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:34.036731 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 02:55:34.036829 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:55:34.045361 19958 raft_consensus.cc:1261] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1 [term 1 FOLLOWER]: Refusing update from remote peer 30411bca67454363a8e3e4682d41e58e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:34.045360 19826 raft_consensus.cc:1261] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2 [term 1 FOLLOWER]: Refusing update from remote peer 30411bca67454363a8e3e4682d41e58e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:34.046546 20032 consensus_queue.cc:1035] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4704cff541e146169860dc7de666f6c2" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 38257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:34.046932 20027 consensus_queue.cc:1035] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36f02192169449c9f23d1ffb7b4b7c1" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 35339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:55:34.052088 19739 tablet.cc:2369] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:34.072124 19549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:34.072453 19549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:34.076186 19549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d5832102cfb44bc9de5215046e78f3f: Sending DeleteTablet for 3 replicas of tablet 892fbb9f08ec4990abdb610bc7a72262
I20260929 02:55:34.077302 19806 tablet_service.cc:1514] Processing DeleteTablet for tablet 892fbb9f08ec4990abdb610bc7a72262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:34 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:36996
I20260929 02:55:34.077308 19673 tablet_service.cc:1514] Processing DeleteTablet for tablet 892fbb9f08ec4990abdb610bc7a72262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:34 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:39848
I20260929 02:55:34.077319 19938 tablet_service.cc:1514] Processing DeleteTablet for tablet 892fbb9f08ec4990abdb610bc7a72262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:34 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:33414
I20260929 02:55:34.079545 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 19611
I20260929 02:55:34.088689 20046 tablet_replica.cc:330] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: stopping tablet replica
I20260929 02:55:34.089737 20046 raft_consensus.cc:2229] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:34.090997 20046 raft_consensus.cc:2258] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:34.092461 20047 tablet_replica.cc:330] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: stopping tablet replica
I20260929 02:55:34.092927 20047 raft_consensus.cc:2229] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:34.093637 20047 raft_consensus.cc:2258] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:34.094354 20046 ts_tablet_manager.cc:1905] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:34.095631 20047 ts_tablet_manager.cc:1905] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:55:34 dist-test-slave-q5f0 krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:34.101558 20046 ts_tablet_manager.cc:1918] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:34.101861 20045 tablet_replica.cc:330] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: stopping tablet replica
I20260929 02:55:34.102416 20045 raft_consensus.cc:2229] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:34.103041 20045 raft_consensus.cc:2258] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:34.103396 20047 ts_tablet_manager.cc:1918] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:34.103636 20047 log.cc:1196] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/892fbb9f08ec4990abdb610bc7a72262
I20260929 02:55:34.104202 20047 ts_tablet_manager.cc:1939] T 892fbb9f08ec4990abdb610bc7a72262 P 30411bca67454363a8e3e4682d41e58e: Deleting consensus metadata
I20260929 02:55:34.105610 20045 ts_tablet_manager.cc:1905] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:34.106144 19534 catalog_manager.cc:4841] TS 30411bca67454363a8e3e4682d41e58e (127.15.18.65:42513): tablet 892fbb9f08ec4990abdb610bc7a72262 (table test-table [id=91b26eef135a4a8b91dea1a1c628387c]) successfully deleted
I20260929 02:55:34.106561 20046 log.cc:1196] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/892fbb9f08ec4990abdb610bc7a72262
I20260929 02:55:34.107118 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:34.081066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.15.18.65:42513)
0929 02:55:34.081270 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:34.081278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:34.084055 (+ 2777us) server_negotiation.cc:407] Connection header received
0929 02:55:34.084249 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:34.084254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:34.084402 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:34.084513 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:34.090023 (+ 5510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.091312 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.093064 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.093928 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.094027 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:34.099260 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:34.099274 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:34.099302 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:34.099359 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:34.103580 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:34.104594 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:34.104604 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:34.104607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:34.104790 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:34.105431 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:34.105436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:34.105439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:34.105873 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:34.106150 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:34.106432 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 02:55:34.106651 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:55:34.107812 19733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:34.107975 20046 ts_tablet_manager.cc:1939] T 892fbb9f08ec4990abdb610bc7a72262 P c36f02192169449c9f23d1ffb7b4b7c1: Deleting consensus metadata
I20260929 02:55:34.109944 19535 catalog_manager.cc:4841] TS c36f02192169449c9f23d1ffb7b4b7c1 (127.15.18.67:35339): tablet 892fbb9f08ec4990abdb610bc7a72262 (table test-table [id=91b26eef135a4a8b91dea1a1c628387c]) successfully deleted
I20260929 02:55:34.113407 20045 ts_tablet_manager.cc:1918] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:34.113567 20045 log.cc:1196] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/892fbb9f08ec4990abdb610bc7a72262
I20260929 02:55:34.114049 20045 ts_tablet_manager.cc:1939] T 892fbb9f08ec4990abdb610bc7a72262 P 4704cff541e146169860dc7de666f6c2: Deleting consensus metadata
I20260929 02:55:34.115795 19537 catalog_manager.cc:4841] TS 4704cff541e146169860dc7de666f6c2 (127.15.18.66:38257): tablet 892fbb9f08ec4990abdb610bc7a72262 (table test-table [id=91b26eef135a4a8b91dea1a1c628387c]) successfully deleted
I20260929 02:55:34.206699 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 19611
I20260929 02:55:34.220743 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 19744
Sep 29 02:55:34 dist-test-slave-q5f0 krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:55:34.235826 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:34.221508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.15.18.66:38257)
0929 02:55:34.221687 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:34.221692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:34.221712 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:55:34.221857 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:34.221862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:34.221955 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:34.222037 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:34.223228 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.224082 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.225445 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.225836 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.225951 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:34.230121 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:34.230137 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:34.230153 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:34.230198 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:34.233020 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:34.233744 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:34.233752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:34.233754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:34.233902 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:34.234282 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:34.234286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:34.234288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:34.234668 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:34.234964 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:34.235261 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 02:55:34.235471 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:55:34.236500 19866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:34.335045 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 19744
I20260929 02:55:34.349072 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 19876
Sep 29 02:55:34 dist-test-slave-q5f0 krb5kdc[19506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:55:34.363153 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:34.349830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.15.18.67:35339)
0929 02:55:34.349987 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:34.349991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:34.350010 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:55:34.350098 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:34.350102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:34.350172 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:34.350256 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:34.351491 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.352252 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.353576 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:34.353943 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:34.354029 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:34.358058 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:34.358074 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:34.358090 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:34.358125 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:34.360806 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:34.361409 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:34.361416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:34.361419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:34.361548 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:34.361866 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:34.361869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:34.361871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:34.362201 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:34.362419 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:34.362729 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 02:55:34.362910 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:55:34.363902 19998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:34.451686 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 19876
I20260929 02:55:34.464157 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 19519
I20260929 02:55:34.465238 19580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:34.571187 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 19519
2026-09-29T02:55:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3930 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T02:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:34Z Disabled control of system clock
I20260929 02:55:34.630511 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:38305
--webserver_interface=127.15.18.126
--webserver_port=0
--builtin_ntp_servers=127.15.18.84:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:55:34.943972 20056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:34.944285 20056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:34.944427 20056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:34.952159 20056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:34.952303 20056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:34.952368 20056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:34.952454 20056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:34.952517 20056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:34.962307 20056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:38305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:34.963730 20056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:34.965760 20056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:34.978119 20062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:34.978359 20064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:34.978102 20061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:34.979393 20056 server_base.cc:1034] running on GCE node
I20260929 02:55:34.980278 20056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:34.981737 20056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:34.982969 20056 hybrid_clock.cc:648] HybridClock initialized: now 1790650534982925 us; error 45 us; skew 500 ppm
I20260929 02:55:34.983289 20056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:55:34.985736 20056 webserver.cc:460] Webserver started at http://127.15.18.126:36069/ using document root <none> and password file <none>
I20260929 02:55:34.986454 20056 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:34.986562 20056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:34.986866 20056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:34.990018 20056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "ddbcefae7cb048e7b3d9c240d59584f2"
format_stamp: "Formatted at 2026-09-29 02:55:34 on dist-test-slave-q5f0"
server_key: "be2267f50ced186ecfb3393dc079276b"
server_key_iv: "c96d29526d26d0527325136a2705eacd"
server_key_version: "encryptionkey@0"
I20260929 02:55:34.990849 20056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "ddbcefae7cb048e7b3d9c240d59584f2"
format_stamp: "Formatted at 2026-09-29 02:55:34 on dist-test-slave-q5f0"
server_key: "be2267f50ced186ecfb3393dc079276b"
server_key_iv: "c96d29526d26d0527325136a2705eacd"
server_key_version: "encryptionkey@0"
I20260929 02:55:34.996438 20056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:55:35.000254 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:35.001902 20056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:35.002079 20056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "ddbcefae7cb048e7b3d9c240d59584f2"
format_stamp: "Formatted at 2026-09-29 02:55:34 on dist-test-slave-q5f0"
server_key: "be2267f50ced186ecfb3393dc079276b"
server_key_iv: "c96d29526d26d0527325136a2705eacd"
server_key_version: "encryptionkey@0"
I20260929 02:55:35.002307 20056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:35.008312 20056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:35.009079 20056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:35.009351 20056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:35.026665 20056 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:38305
I20260929 02:55:35.026683 20121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:38305 every 8 connection(s)
I20260929 02:55:35.028034 20056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:35.031788 20122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:35.038336 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 20056
I20260929 02:55:35.038496 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:35.039116 15433 external_mini_cluster.cc:1428] Setting key 94084ddf26c73244e5991317ea530d41
I20260929 02:55:35.042320 20122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2: Bootstrap starting.
I20260929 02:55:35.045853 20122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:35.046939 20122 log.cc:824] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:35.049969 20122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2: No bootstrap required, opened a new log
I20260929 02:55:35.053170 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:35.041160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.15.18.126:38305)
0929 02:55:35.042149 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:35.042161 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:35.042205 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:55:35.043981 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:35.044006 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:35.044470 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:35.044835 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:35.051322 (+ 6487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:35.051329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:35.051351 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:35.051371 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:35.051445 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:35.051465 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:35.051718 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:35.051874 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 02:55:35.052165 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":332,"threads_started":1}
I20260929 02:55:35.057868 20122 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbcefae7cb048e7b3d9c240d59584f2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38305 } }
I20260929 02:55:35.058215 20122 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:35.058308 20122 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbcefae7cb048e7b3d9c240d59584f2, State: Initialized, Role: FOLLOWER
I20260929 02:55:35.058856 20122 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbcefae7cb048e7b3d9c240d59584f2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38305 } }
I20260929 02:55:35.059063 20122 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:35.059178 20122 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:35.059330 20122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:35.061692 20122 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbcefae7cb048e7b3d9c240d59584f2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38305 } }
I20260929 02:55:35.062242 20122 leader_election.cc:302] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddbcefae7cb048e7b3d9c240d59584f2; no voters:
I20260929 02:55:35.062847 20122 leader_election.cc:288] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:35.062971 20127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:35.063594 20127 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [term 1 LEADER]: Becoming Leader. State: Replica: ddbcefae7cb048e7b3d9c240d59584f2, State: Running, Role: LEADER
I20260929 02:55:35.064220 20127 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbcefae7cb048e7b3d9c240d59584f2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38305 } }
I20260929 02:55:35.065068 20122 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:35.067137 20128 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddbcefae7cb048e7b3d9c240d59584f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbcefae7cb048e7b3d9c240d59584f2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38305 } } }
I20260929 02:55:35.067240 20129 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddbcefae7cb048e7b3d9c240d59584f2. Latest consensus state: current_term: 1 leader_uuid: "ddbcefae7cb048e7b3d9c240d59584f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbcefae7cb048e7b3d9c240d59584f2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38305 } } }
I20260929 02:55:35.067566 20128 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:35.067628 20129 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:35.072237 20134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:35.078023 20134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:35.087405 20134 catalog_manager.cc:1347] Generated new cluster ID: 1c50ca252a414e6fbdb8b1db998d37cd
I20260929 02:55:35.087558 20134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:35.108635 20134 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:35.109133 20134 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:35.117010 20134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2: Generated new TSK 0
I20260929 02:55:35.118108 20134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:55:35.126792 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38305
--builtin_ntp_servers=127.15.18.84:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:55:35.460103 20146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:35.460400 20146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:35.460515 20146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:35.467916 20146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:35.468031 20146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:35.468189 20146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:35.477548 20146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:35.478853 20146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:35.480618 20146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:35.493358 20152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:35.493430 20154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:35.493430 20151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:35.494071 20146 server_base.cc:1034] running on GCE node
I20260929 02:55:35.494727 20146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:35.495748 20146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:35.496964 20146 hybrid_clock.cc:648] HybridClock initialized: now 1790650535496900 us; error 61 us; skew 500 ppm
I20260929 02:55:35.497283 20146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:55:35.499603 20146 webserver.cc:460] Webserver started at http://127.15.18.65:36813/ using document root <none> and password file <none>
I20260929 02:55:35.500289 20146 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:35.500391 20146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:35.500713 20146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:35.503479 20146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "072a36b2e1e144b2860e939697b9f5c9"
format_stamp: "Formatted at 2026-09-29 02:55:35 on dist-test-slave-q5f0"
server_key: "59962b9212668bf8f0075ecfc39d6e9b"
server_key_iv: "988c8c0960fd7dced7b18ac38cf98ed7"
server_key_version: "encryptionkey@0"
I20260929 02:55:35.504377 20146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "072a36b2e1e144b2860e939697b9f5c9"
format_stamp: "Formatted at 2026-09-29 02:55:35 on dist-test-slave-q5f0"
server_key: "59962b9212668bf8f0075ecfc39d6e9b"
server_key_iv: "988c8c0960fd7dced7b18ac38cf98ed7"
server_key_version: "encryptionkey@0"
I20260929 02:55:35.509593 20146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:55:35.513033 20160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:35.514349 20146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:55:35.514515 20146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "072a36b2e1e144b2860e939697b9f5c9"
format_stamp: "Formatted at 2026-09-29 02:55:35 on dist-test-slave-q5f0"
server_key: "59962b9212668bf8f0075ecfc39d6e9b"
server_key_iv: "988c8c0960fd7dced7b18ac38cf98ed7"
server_key_version: "encryptionkey@0"
I20260929 02:55:35.514724 20146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:35.520274 20146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:35.521142 20146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:35.521385 20146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:35.522269 20146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:35.523681 20146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:35.523774 20146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:35.523880 20146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:35.523945 20146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:35.553500 20146 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:35739
I20260929 02:55:35.553578 20272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:35739 every 8 connection(s)
I20260929 02:55:35.554790 20146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:35.563127 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:35.557345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:53323 (local address 127.15.18.126:38305)
0929 02:55:35.557750 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:35.557755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:35.559034 (+ 1279us) server_negotiation.cc:407] Connection header received
0929 02:55:35.560730 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:35.560735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:35.560825 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:35.560918 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:35.562330 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:35.562334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:35.562337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:35.562356 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:35.562400 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:35.562413 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:35.562551 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:35.562735 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 02:55:35.562806 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":209,"threads_started":1}
I20260929 02:55:35.564173 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:35.557927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38305 (local address 127.15.18.65:53323)
0929 02:55:35.558826 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:35.558867 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:35.560459 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:35.561180 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:35.561188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:35.561288 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:35.561977 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:55:35.561980 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:35.562024 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:55:35.562133 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:35.562509 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:35.562520 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:35.562529 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:55:35.562852 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 02:55:35.563120 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":253,"threads_started":1}
I20260929 02:55:35.564653 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 20146
I20260929 02:55:35.564854 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:35.565567 15433 external_mini_cluster.cc:1428] Setting key 73bc01b8384ca1d2da2d74e5e9b744b1
I20260929 02:55:35.565805 20273 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38305
I20260929 02:55:35.566124 20273 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:35.566857 20273 heartbeater.cc:511] Master 127.15.18.126:38305 requested a full tablet report, sending...
I20260929 02:55:35.568562 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38305
--builtin_ntp_servers=127.15.18.84:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 02:55:35.569160 20087 ts_manager.cc:194] Registered new tserver with Master: 072a36b2e1e144b2860e939697b9f5c9 (127.15.18.65:35739)
W20260929 02:55:35.883647 20277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:35.883984 20277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:35.884112 20277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:35.891598 20277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:35.891741 20277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:35.891896 20277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:35.901347 20277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:35.902647 20277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:35.904527 20277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:35.917637 20283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:35.917794 20285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:35.917829 20282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:35.921159 20277 server_base.cc:1034] running on GCE node
I20260929 02:55:35.921797 20277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:35.922780 20277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:35.924000 20277 hybrid_clock.cc:648] HybridClock initialized: now 1790650535923948 us; error 37 us; skew 500 ppm
I20260929 02:55:35.924355 20277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:55:35.927032 20277 webserver.cc:460] Webserver started at http://127.15.18.66:45129/ using document root <none> and password file <none>
I20260929 02:55:35.927783 20277 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:35.927883 20277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:35.928182 20277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:35.931005 20277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "ec5ba43127db4e3c9ff86b7d63c5d200"
format_stamp: "Formatted at 2026-09-29 02:55:35 on dist-test-slave-q5f0"
server_key: "71782496cee9e0aa16e666d8c9b4e347"
server_key_iv: "25e72b92a24686dfe8c5eb70c0f3b497"
server_key_version: "encryptionkey@0"
I20260929 02:55:35.931797 20277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "ec5ba43127db4e3c9ff86b7d63c5d200"
format_stamp: "Formatted at 2026-09-29 02:55:35 on dist-test-slave-q5f0"
server_key: "71782496cee9e0aa16e666d8c9b4e347"
server_key_iv: "25e72b92a24686dfe8c5eb70c0f3b497"
server_key_version: "encryptionkey@0"
I20260929 02:55:35.936789 20277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:55:35.940220 20291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:35.941702 20277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:35.941870 20277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "ec5ba43127db4e3c9ff86b7d63c5d200"
format_stamp: "Formatted at 2026-09-29 02:55:35 on dist-test-slave-q5f0"
server_key: "71782496cee9e0aa16e666d8c9b4e347"
server_key_iv: "25e72b92a24686dfe8c5eb70c0f3b497"
server_key_version: "encryptionkey@0"
I20260929 02:55:35.942119 20277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:35.948086 20277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:35.948915 20277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:35.949162 20277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:35.950147 20277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:35.951645 20277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:35.951740 20277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:35.951848 20277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:35.951920 20277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:35.982129 20277 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:37403
I20260929 02:55:35.982165 20403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:37403 every 8 connection(s)
I20260929 02:55:35.983460 20277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:35.987087 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 20277
I20260929 02:55:35.987232 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:35.987709 15433 external_mini_cluster.cc:1428] Setting key 5b520ebce4c3ca803ccc4cf2e39ec96d
I20260929 02:55:35.990345 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38305
--builtin_ntp_servers=127.15.18.84:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 02:55:35.991798 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:35.985991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:45629 (local address 127.15.18.126:38305)
0929 02:55:35.986160 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:35.986165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:35.987775 (+ 1610us) server_negotiation.cc:407] Connection header received
0929 02:55:35.989630 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:35.989636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:35.989698 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:35.989785 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:35.991068 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:35.991074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:35.991079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:35.991097 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:35.991127 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:35.991137 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:35.991253 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:35.991430 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 02:55:35.991490 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:55:35.992725 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:35.986540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38305 (local address 127.15.18.66:45629)
0929 02:55:35.987595 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:35.987646 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:35.989366 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:35.990010 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:35.990019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:35.990116 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:35.990788 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:55:35.990790 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:35.990827 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:55:35.990913 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:35.991216 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:35.991227 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:35.991235 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:55:35.991500 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 02:55:35.991786 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":298,"threads_started":1}
I20260929 02:55:35.994068 20404 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38305
I20260929 02:55:35.994396 20404 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:35.994995 20404 heartbeater.cc:511] Master 127.15.18.126:38305 requested a full tablet report, sending...
I20260929 02:55:35.996521 20087 ts_manager.cc:194] Registered new tserver with Master: ec5ba43127db4e3c9ff86b7d63c5d200 (127.15.18.66:37403)
W20260929 02:55:36.312341 20407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:36.312608 20407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:36.312716 20407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:36.320191 20407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:36.320307 20407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:36.320465 20407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:36.330009 20407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:36.331357 20407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:36.333257 20407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:36.346117 20415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:36.346093 20413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:36.346407 20412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:36.346915 20407 server_base.cc:1034] running on GCE node
I20260929 02:55:36.347570 20407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:36.348533 20407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:36.349737 20407 hybrid_clock.cc:648] HybridClock initialized: now 1790650536349688 us; error 42 us; skew 500 ppm
I20260929 02:55:36.350093 20407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:55:36.352459 20407 webserver.cc:460] Webserver started at http://127.15.18.67:33595/ using document root <none> and password file <none>
I20260929 02:55:36.353195 20407 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:36.353291 20407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:36.353614 20407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:36.356523 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "e223e516dd4f4e5fbf5cfd6797869553"
format_stamp: "Formatted at 2026-09-29 02:55:36 on dist-test-slave-q5f0"
server_key: "d3ace3d96e9aa545d6efa0c48292f85c"
server_key_iv: "c03a8c7fb664e8f5dd7b792bde081632"
server_key_version: "encryptionkey@0"
I20260929 02:55:36.357266 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "e223e516dd4f4e5fbf5cfd6797869553"
format_stamp: "Formatted at 2026-09-29 02:55:36 on dist-test-slave-q5f0"
server_key: "d3ace3d96e9aa545d6efa0c48292f85c"
server_key_iv: "c03a8c7fb664e8f5dd7b792bde081632"
server_key_version: "encryptionkey@0"
I20260929 02:55:36.362332 20407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:36.365770 20421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:36.367112 20407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 02:55:36.367254 20407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "e223e516dd4f4e5fbf5cfd6797869553"
format_stamp: "Formatted at 2026-09-29 02:55:36 on dist-test-slave-q5f0"
server_key: "d3ace3d96e9aa545d6efa0c48292f85c"
server_key_iv: "c03a8c7fb664e8f5dd7b792bde081632"
server_key_version: "encryptionkey@0"
I20260929 02:55:36.367447 20407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:36.372941 20407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:36.373744 20407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:36.373983 20407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:36.374837 20407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:36.376260 20407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:36.376339 20407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:36.376420 20407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:36.376472 20407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:36.404711 20407 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:34223
I20260929 02:55:36.404757 20533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:34223 every 8 connection(s)
I20260929 02:55:36.406127 20407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:36.409334 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 20407
I20260929 02:55:36.409526 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:36.410099 15433 external_mini_cluster.cc:1428] Setting key f986c9f344b08f6ffcc58aeea8b8d276
I20260929 02:55:36.415030 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.408834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:35439 (local address 127.15.18.126:38305)
0929 02:55:36.409036 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.409040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.410806 (+ 1766us) server_negotiation.cc:407] Connection header received
0929 02:55:36.412890 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.412894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.412966 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.413082 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.414436 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.414441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.414444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.414459 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.414489 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.414500 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.414589 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.414760 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.414809 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:55:36.416051 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.409495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38305 (local address 127.15.18.67:35439)
0929 02:55:36.410580 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.410637 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:36.412577 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:36.413350 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:36.413360 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:36.413464 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:36.414135 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:55:36.414138 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:36.414178 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:55:36.414268 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:36.414576 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:36.414582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:36.414587 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:55:36.414799 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 02:55:36.415031 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":328,"threads_started":1}
I20260929 02:55:36.417328 20534 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38305
I20260929 02:55:36.417603 20534 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:36.418222 20534 heartbeater.cc:511] Master 127.15.18.126:38305 requested a full tablet report, sending...
I20260929 02:55:36.419631 20087 ts_manager.cc:194] Registered new tserver with Master: e223e516dd4f4e5fbf5cfd6797869553 (127.15.18.67:34223)
I20260929 02:55:36.425546 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:55:36.432538 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.429842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.15.18.126:38305)
0929 02:55:36.430018 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.430025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.430500 (+ 475us) server_negotiation.cc:407] Connection header received
0929 02:55:36.430749 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.430755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.430823 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.430916 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.431933 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.431939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.431944 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.431963 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.432004 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.432019 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.432145 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.432189 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.432248 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:55:36.437028 20087 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:36.443645 20087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:36.477538 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.455973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.15.18.65:35739)
0929 02:55:36.472345 (+ 16372us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.472355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.473954 (+ 1599us) server_negotiation.cc:407] Connection header received
0929 02:55:36.474182 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.474190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.474401 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.474517 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.476607 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.476617 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.476632 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.476659 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.476720 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.476743 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.476886 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.477054 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.477164 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16168,"thread_start_us":16018,"threads_started":1}
I20260929 02:55:36.477592 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.455321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:35739 (local address 127.0.0.1:42484)
0929 02:55:36.473830 (+ 18509us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.473847 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:36.474038 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:36.475277 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:36.475284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:36.475384 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:36.476274 (+ 890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:55:36.476278 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:36.476308 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:55:36.476407 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:36.476850 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:36.476863 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:36.476870 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:55:36.477039 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 02:55:36.477139 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18374,"thread_start_us":19604,"threads_started":1}
I20260929 02:55:36.477839 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.456086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.15.18.67:34223)
0929 02:55:36.471822 (+ 15736us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.471834 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.475275 (+ 3441us) server_negotiation.cc:407] Connection header received
0929 02:55:36.475506 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.475539 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.475758 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.475920 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.476951 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.476958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.476969 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.476986 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.477035 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.477053 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.477209 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.477415 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.477510 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15257,"thread_start_us":15496,"threads_started":1}
I20260929 02:55:36.478634 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.455616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:34223 (local address 127.0.0.1:34078)
0929 02:55:36.475162 (+ 19546us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.475178 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:36.475349 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:36.475908 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:36.475912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:36.475933 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:36.476491 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:55:36.476495 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:36.476512 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:55:36.476555 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:36.477263 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:36.477268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:36.477277 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:55:36.478144 (+ 867us) client_negotiation.cc:240] Negotiation successful
0929 02:55:36.478200 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19437,"thread_start_us":16908,"threads_started":1}
I20260929 02:55:36.478703 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.455067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:37403 (local address 127.0.0.1:52288)
0929 02:55:36.473079 (+ 18012us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.473129 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:36.473299 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:36.475394 (+ 2095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:36.475399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:36.475423 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:36.476518 (+ 1095us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:55:36.476521 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:36.476533 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:55:36.476561 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:36.478068 (+ 1507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:36.478071 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:36.478076 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:55:36.478213 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 02:55:36.478266 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17626,"thread_start_us":19515,"threads_started":1}
I20260929 02:55:36.479461 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.455484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.15.18.66:37403)
0929 02:55:36.473671 (+ 18187us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.473679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.473711 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:55:36.473783 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.473791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.474081 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.474229 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.476733 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.476740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.476751 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.476776 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.476831 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.476852 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.477010 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.479036 (+ 2026us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.479117 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18020,"thread_start_us":17863,"threads_started":1}
I20260929 02:55:36.481993 20208 tablet_service.cc:1467] Processing CreateTablet for tablet 6a390951bdfb4c67b12999ddde84b2da (DEFAULT_TABLE table=test-table [id=35857a7dfc7941858022212a01f89cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:36.482184 20469 tablet_service.cc:1467] Processing CreateTablet for tablet 6a390951bdfb4c67b12999ddde84b2da (DEFAULT_TABLE table=test-table [id=35857a7dfc7941858022212a01f89cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:36.482784 20339 tablet_service.cc:1467] Processing CreateTablet for tablet 6a390951bdfb4c67b12999ddde84b2da (DEFAULT_TABLE table=test-table [id=35857a7dfc7941858022212a01f89cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:36.483453 20208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a390951bdfb4c67b12999ddde84b2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:36.483601 20469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a390951bdfb4c67b12999ddde84b2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:36.484078 20339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a390951bdfb4c67b12999ddde84b2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:36.495652 20552 tablet_bootstrap.cc:492] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9: Bootstrap starting.
I20260929 02:55:36.499047 20553 tablet_bootstrap.cc:492] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: Bootstrap starting.
I20260929 02:55:36.499341 20552 tablet_bootstrap.cc:654] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:36.499446 20554 tablet_bootstrap.cc:492] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: Bootstrap starting.
I20260929 02:55:36.500734 20552 log.cc:824] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:36.502553 20553 tablet_bootstrap.cc:654] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:36.502573 20554 tablet_bootstrap.cc:654] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:36.503499 20554 log.cc:824] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:36.503711 20553 log.cc:824] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:36.504434 20552 tablet_bootstrap.cc:492] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9: No bootstrap required, opened a new log
I20260929 02:55:36.504695 20552 ts_tablet_manager.cc:1397] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 02:55:36.506222 20553 tablet_bootstrap.cc:492] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: No bootstrap required, opened a new log
I20260929 02:55:36.506418 20554 tablet_bootstrap.cc:492] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: No bootstrap required, opened a new log
I20260929 02:55:36.506561 20553 ts_tablet_manager.cc:1397] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:55:36.506666 20554 ts_tablet_manager.cc:1397] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 02:55:36.514110 20553 raft_consensus.cc:348] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 } } peers { permanent_uuid: "072a36b2e1e144b2860e939697b9f5c9" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 35739 } } peers { permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 } }
I20260929 02:55:36.514498 20553 raft_consensus.cc:374] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:36.514611 20553 raft_consensus.cc:729] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e223e516dd4f4e5fbf5cfd6797869553, State: Initialized, Role: FOLLOWER
I20260929 02:55:36.514917 20552 raft_consensus.cc:348] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 } } peers { permanent_uuid: "072a36b2e1e144b2860e939697b9f5c9" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 35739 } } peers { permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 } }
I20260929 02:55:36.515177 20553 consensus_queue.cc:260] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 } } peers { permanent_uuid: "072a36b2e1e144b2860e939697b9f5c9" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 35739 } } peers { permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 } }
I20260929 02:55:36.515339 20552 raft_consensus.cc:374] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:36.515422 20552 raft_consensus.cc:729] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072a36b2e1e144b2860e939697b9f5c9, State: Initialized, Role: FOLLOWER
I20260929 02:55:36.516048 20552 consensus_queue.cc:260] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 } } peers { permanent_uuid: "072a36b2e1e144b2860e939697b9f5c9" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 35739 } } peers { permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 } }
I20260929 02:55:36.516441 20534 heartbeater.cc:503] Master 127.15.18.126:38305 was elected leader, sending a full tablet report...
I20260929 02:55:36.516667 20554 raft_consensus.cc:348] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 } } peers { permanent_uuid: "072a36b2e1e144b2860e939697b9f5c9" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 35739 } } peers { permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 } }
I20260929 02:55:36.517109 20553 ts_tablet_manager.cc:1428] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260929 02:55:36.517167 20554 raft_consensus.cc:374] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:36.517336 20554 raft_consensus.cc:729] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5ba43127db4e3c9ff86b7d63c5d200, State: Initialized, Role: FOLLOWER
I20260929 02:55:36.518064 20273 heartbeater.cc:503] Master 127.15.18.126:38305 was elected leader, sending a full tablet report...
I20260929 02:55:36.518002 20554 consensus_queue.cc:260] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 } } peers { permanent_uuid: "072a36b2e1e144b2860e939697b9f5c9" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 35739 } } peers { permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 } }
I20260929 02:55:36.518745 20552 ts_tablet_manager.cc:1428] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 02:55:36.519645 20404 heartbeater.cc:503] Master 127.15.18.126:38305 was elected leader, sending a full tablet report...
I20260929 02:55:36.520295 20554 ts_tablet_manager.cc:1428] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260929 02:55:36.556821 20274 tablet.cc:2369] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:36.632019 20559 raft_consensus.cc:482] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:36.632364 20559 raft_consensus.cc:504] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 } } peers { permanent_uuid: "072a36b2e1e144b2860e939697b9f5c9" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 35739 } } peers { permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 } }
I20260929 02:55:36.634021 20559 leader_election.cc:288] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec5ba43127db4e3c9ff86b7d63c5d200 (127.15.18.66:37403), e223e516dd4f4e5fbf5cfd6797869553 (127.15.18.67:34223)
I20260929 02:55:36.637126 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.634406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:47793 (local address 127.15.18.66:37403)
0929 02:55:36.634617 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.634624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.635269 (+ 645us) server_negotiation.cc:407] Connection header received
0929 02:55:36.635444 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.635448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.635498 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.635611 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.636557 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.636562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.636565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.636582 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.636613 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.636624 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.636737 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.636808 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.636872 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 02:55:36.637209 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.634571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:46161 (local address 127.15.18.67:34223)
0929 02:55:36.634746 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.634752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.635445 (+ 693us) server_negotiation.cc:407] Connection header received
0929 02:55:36.635607 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.635611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.635658 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.635737 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.636664 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.636669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.636671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.636687 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.636715 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.636725 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.636829 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.636919 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.636974 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:55:36.637422 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.634284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:37403 (local address 127.15.18.65:47793)
0929 02:55:36.635149 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.635172 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:36.635319 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:36.635641 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:36.635645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:36.635668 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:36.636334 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:55:36.636337 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:36.636352 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:55:36.636392 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:36.636710 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:36.636715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:36.636719 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 02:55:36.636888 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 02:55:36.636942 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":518,"threads_started":1}
I20260929 02:55:36.637454 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.634490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:34223 (local address 127.15.18.65:46161)
0929 02:55:36.635338 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.635352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:36.635455 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:36.635791 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:36.635798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:36.635820 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:36.636472 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:55:36.636474 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:36.636486 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:55:36.636526 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:36.636810 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:36.636814 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:36.636819 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:55:36.636963 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 02:55:36.637011 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":495,"threads_started":1}
I20260929 02:55:36.638305 20359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a390951bdfb4c67b12999ddde84b2da" candidate_uuid: "072a36b2e1e144b2860e939697b9f5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" is_pre_election: true
I20260929 02:55:36.638319 20489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a390951bdfb4c67b12999ddde84b2da" candidate_uuid: "072a36b2e1e144b2860e939697b9f5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e223e516dd4f4e5fbf5cfd6797869553" is_pre_election: true
I20260929 02:55:36.638761 20489 raft_consensus.cc:2454] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 072a36b2e1e144b2860e939697b9f5c9 in term 0.
I20260929 02:55:36.638760 20359 raft_consensus.cc:2454] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 072a36b2e1e144b2860e939697b9f5c9 in term 0.
I20260929 02:55:36.639354 20161 leader_election.cc:302] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072a36b2e1e144b2860e939697b9f5c9, e223e516dd4f4e5fbf5cfd6797869553; no voters:
I20260929 02:55:36.639729 20559 raft_consensus.cc:2781] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:36.639878 20559 raft_consensus.cc:482] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:36.639972 20559 raft_consensus.cc:3037] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:36.642037 20559 raft_consensus.cc:504] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 } } peers { permanent_uuid: "072a36b2e1e144b2860e939697b9f5c9" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 35739 } } peers { permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 } }
I20260929 02:55:36.642609 20559 leader_election.cc:288] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [CANDIDATE]: Term 1 election: Requested vote from peers ec5ba43127db4e3c9ff86b7d63c5d200 (127.15.18.66:37403), e223e516dd4f4e5fbf5cfd6797869553 (127.15.18.67:34223)
I20260929 02:55:36.643115 20359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a390951bdfb4c67b12999ddde84b2da" candidate_uuid: "072a36b2e1e144b2860e939697b9f5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200"
I20260929 02:55:36.643173 20489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a390951bdfb4c67b12999ddde84b2da" candidate_uuid: "072a36b2e1e144b2860e939697b9f5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e223e516dd4f4e5fbf5cfd6797869553"
I20260929 02:55:36.643337 20359 raft_consensus.cc:3037] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:36.643378 20489 raft_consensus.cc:3037] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:36.646574 20489 raft_consensus.cc:2454] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 072a36b2e1e144b2860e939697b9f5c9 in term 1.
I20260929 02:55:36.646574 20359 raft_consensus.cc:2454] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 072a36b2e1e144b2860e939697b9f5c9 in term 1.
I20260929 02:55:36.647009 20161 leader_election.cc:302] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072a36b2e1e144b2860e939697b9f5c9, e223e516dd4f4e5fbf5cfd6797869553; no voters:
I20260929 02:55:36.647296 20559 raft_consensus.cc:2781] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:36.647766 20559 raft_consensus.cc:686] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 1 LEADER]: Becoming Leader. State: Replica: 072a36b2e1e144b2860e939697b9f5c9, State: Running, Role: LEADER
I20260929 02:55:36.648291 20559 consensus_queue.cc:237] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 } } peers { permanent_uuid: "072a36b2e1e144b2860e939697b9f5c9" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 35739 } } peers { permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 } }
I20260929 02:55:36.652779 20087 catalog_manager.cc:5495] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 072a36b2e1e144b2860e939697b9f5c9 (127.15.18.65). New cstate: current_term: 1 leader_uuid: "072a36b2e1e144b2860e939697b9f5c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "072a36b2e1e144b2860e939697b9f5c9" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 35739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 } health_report { overall_health: UNKNOWN } } }
W20260929 02:55:36.657771 20535 tablet.cc:2369] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:36.670138 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.668220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.15.18.65:35739)
0929 02:55:36.668368 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.668372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.668448 (+ 76us) server_negotiation.cc:407] Connection header received
0929 02:55:36.668617 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.668621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.668697 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.668782 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.669627 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.669630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.669633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.669644 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.669669 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.669678 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.669759 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.669858 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.669905 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:55:36.678853 20359 raft_consensus.cc:1261] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200 [term 1 FOLLOWER]: Refusing update from remote peer 072a36b2e1e144b2860e939697b9f5c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:36.678853 20489 raft_consensus.cc:1261] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553 [term 1 FOLLOWER]: Refusing update from remote peer 072a36b2e1e144b2860e939697b9f5c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:36.679988 20559 consensus_queue.cc:1035] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec5ba43127db4e3c9ff86b7d63c5d200" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 37403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:36.680330 20563 consensus_queue.cc:1035] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e223e516dd4f4e5fbf5cfd6797869553" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:36.689881 20566 mvcc.cc:205] Tried to move back new op lower bound from 7334504598225252352 to 7334504598115196928. Current Snapshot: MvccSnapshot[applied={T|T < 7334504598225252352}]
I20260929 02:55:36.691300 20567 mvcc.cc:205] Tried to move back new op lower bound from 7334504598225252352 to 7334504598115196928. Current Snapshot: MvccSnapshot[applied={T|T < 7334504598225252352}]
I20260929 02:55:36.701368 20568 mvcc.cc:205] Tried to move back new op lower bound from 7334504598225252352 to 7334504598115196928. Current Snapshot: MvccSnapshot[applied={T|T < 7334504598225252352}]
I20260929 02:55:36.714745 20085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:36.715059 20085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:36.718899 20085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddbcefae7cb048e7b3d9c240d59584f2: Sending DeleteTablet for 3 replicas of tablet 6a390951bdfb4c67b12999ddde84b2da
I20260929 02:55:36.719957 20208 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a390951bdfb4c67b12999ddde84b2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:36 UTC) from {username='slave'} at 127.0.0.1:42484
I20260929 02:55:36.720009 20339 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a390951bdfb4c67b12999ddde84b2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:36 UTC) from {username='slave'} at 127.0.0.1:52288
I20260929 02:55:36.720569 20469 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a390951bdfb4c67b12999ddde84b2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:36 UTC) from {username='slave'} at 127.0.0.1:34078
I20260929 02:55:36.721612 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 20146
I20260929 02:55:36.735617 20578 tablet_replica.cc:330] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: stopping tablet replica
I20260929 02:55:36.735873 20577 tablet_replica.cc:330] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: stopping tablet replica
I20260929 02:55:36.736101 20578 raft_consensus.cc:2229] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:36.736585 20576 tablet_replica.cc:330] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9: stopping tablet replica
I20260929 02:55:36.736591 20577 raft_consensus.cc:2229] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:36.736815 20578 raft_consensus.cc:2258] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:36.737027 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.723335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.15.18.65:35739)
0929 02:55:36.723869 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.723876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.733412 (+ 9536us) server_negotiation.cc:407] Connection header received
0929 02:55:36.733934 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.733942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.734017 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.734093 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.736210 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.736216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.736221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.736251 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.736290 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.736307 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.736447 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.736692 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.736761 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408}
I20260929 02:55:36.737142 20577 raft_consensus.cc:2258] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:36.737033 20576 raft_consensus.cc:2229] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:36.737994 20576 raft_consensus.cc:2258] T 6a390951bdfb4c67b12999ddde84b2da P 072a36b2e1e144b2860e939697b9f5c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:36.738070 20268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:36.739311 20578 ts_tablet_manager.cc:1905] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:36.741041 20577 ts_tablet_manager.cc:1905] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:36.744359 20578 ts_tablet_manager.cc:1918] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:36.744551 20578 log.cc:1196] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/6a390951bdfb4c67b12999ddde84b2da
I20260929 02:55:36.745033 20578 ts_tablet_manager.cc:1939] T 6a390951bdfb4c67b12999ddde84b2da P e223e516dd4f4e5fbf5cfd6797869553: Deleting consensus metadata
I20260929 02:55:36.747059 20071 catalog_manager.cc:4841] TS e223e516dd4f4e5fbf5cfd6797869553 (127.15.18.67:34223): tablet 6a390951bdfb4c67b12999ddde84b2da (table test-table [id=35857a7dfc7941858022212a01f89cc1]) successfully deleted
I20260929 02:55:36.748302 20577 ts_tablet_manager.cc:1918] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:36.748443 20577 log.cc:1196] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/6a390951bdfb4c67b12999ddde84b2da
I20260929 02:55:36.748831 20577 ts_tablet_manager.cc:1939] T 6a390951bdfb4c67b12999ddde84b2da P ec5ba43127db4e3c9ff86b7d63c5d200: Deleting consensus metadata
I20260929 02:55:36.750216 20072 catalog_manager.cc:4841] TS ec5ba43127db4e3c9ff86b7d63c5d200 (127.15.18.66:37403): tablet 6a390951bdfb4c67b12999ddde84b2da (table test-table [id=35857a7dfc7941858022212a01f89cc1]) successfully deleted
I20260929 02:55:36.834844 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 20146
W20260929 02:55:36.847941 20073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.18.65:35739 (error 108)
I20260929 02:55:36.848599 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 20277
I20260929 02:55:36.849710 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.849226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:35739 (local address 127.0.0.1:42518)
0929 02:55:36.849471 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.849576 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260929 02:55:36.849975 20073 catalog_manager.cc:4584] TS 072a36b2e1e144b2860e939697b9f5c9 (127.15.18.65:35739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a390951bdfb4c67b12999ddde84b2da: Network error: Client connection negotiation failed: client connection to 127.15.18.65:35739: connect: Connection refused (error 111)
I20260929 02:55:36.851334 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.849274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.15.18.66:37403)
0929 02:55:36.849508 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.849513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.849531 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:36.849663 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.849668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.849800 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.849891 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.850755 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.850762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.850767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.850785 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.850818 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.850829 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.850934 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.851019 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.851075 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 02:55:36.852036 20399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:36.903661 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.903062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:35739 (local address 127.0.0.1:42520)
0929 02:55:36.903317 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.903429 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 02:55:36.904785 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.904414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:35739 (local address 127.0.0.1:42532)
0929 02:55:36.904583 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.904661 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 02:55:36.939031 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 20277
I20260929 02:55:36.951795 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 20407
I20260929 02:55:36.954433 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.952543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.15.18.67:34223)
0929 02:55:36.952697 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:36.952701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:36.952720 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:55:36.952894 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:36.952897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:36.952976 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:36.953065 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:36.953945 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:36.953949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:36.953952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:36.953966 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:36.953995 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:36.954005 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:36.954102 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:36.954159 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 02:55:36.954206 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:55:36.955065 20529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:36.965772 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.965327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:35739 (local address 127.0.0.1:42546)
0929 02:55:36.965532 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.965617 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 02:55:36.966681 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:36.966361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:35739 (local address 127.0.0.1:42556)
0929 02:55:36.966498 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:36.966569 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 02:55:37.042008 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 20407
I20260929 02:55:37.047175 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:37.046736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:35739 (local address 127.0.0.1:42564)
0929 02:55:37.046950 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:37.047047 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 02:55:37.048421 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:37.047974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:35739 (local address 127.0.0.1:42572)
0929 02:55:37.048216 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:37.048294 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:35739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 02:55:37.055032 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 20056
I20260929 02:55:37.056113 20117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:37.160434 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 20056
2026-09-29T02:55:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2588 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 02:55:37.197001 15433 mini_oidc.cc:152] Starting JWKS server
I20260929 02:55:37.199559 15433 webserver.cc:460] Webserver started at https://127.0.0.1:42495/ using document root <none> and password file <none>
I20260929 02:55:37.199858 15433 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:55:37.200465 15433 webserver.cc:460] Webserver started at http://127.0.0.1:42039/ using document root <none> and password file <none>
2026-09-29T02:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:37Z Disabled control of system clock
I20260929 02:55:37.225778 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:44597
--webserver_interface=127.15.18.126
--webserver_port=0
--builtin_ntp_servers=127.15.18.84:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:44597
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42495/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/testchainca.cert with env {}
W20260929 02:55:37.545977 20590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:37.546276 20590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:37.546394 20590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:37.558212 20590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:37.558496 20590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:37.558563 20590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:37.558643 20590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:37.558705 20590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:55:37.558794 20590 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:55:37.558851 20590 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42495/jwks.json
I20260929 02:55:37.573496 20590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:44597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:44597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42495/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:37.575278 20590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:37.577085 20590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:37.591336 20596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:37.591282 20595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:37.591367 20598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:37.592587 20590 server_base.cc:1034] running on GCE node
I20260929 02:55:37.593452 20590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:37.595005 20590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:37.596247 20590 hybrid_clock.cc:648] HybridClock initialized: now 1790650537596184 us; error 46 us; skew 500 ppm
I20260929 02:55:37.596601 20590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:55:37.598874 20590 webserver.cc:460] Webserver started at http://127.15.18.126:38271/ using document root <none> and password file <none>
I20260929 02:55:37.599617 20590 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:37.599732 20590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:37.600057 20590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:37.603051 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "c3eb2d69100f49e1afa562f899f0bd08"
format_stamp: "Formatted at 2026-09-29 02:55:37 on dist-test-slave-q5f0"
server_key: "b7a25593b4cc95c5ad935fac85044b09"
server_key_iv: "ef9b0a9958ef0dfd90c640f7dc98df9a"
server_key_version: "encryptionkey@0"
I20260929 02:55:37.603997 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "c3eb2d69100f49e1afa562f899f0bd08"
format_stamp: "Formatted at 2026-09-29 02:55:37 on dist-test-slave-q5f0"
server_key: "b7a25593b4cc95c5ad935fac85044b09"
server_key_iv: "ef9b0a9958ef0dfd90c640f7dc98df9a"
server_key_version: "encryptionkey@0"
I20260929 02:55:37.609434 20590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:55:37.613436 20604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:37.614966 20590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:55:37.615151 20590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "c3eb2d69100f49e1afa562f899f0bd08"
format_stamp: "Formatted at 2026-09-29 02:55:37 on dist-test-slave-q5f0"
server_key: "b7a25593b4cc95c5ad935fac85044b09"
server_key_iv: "ef9b0a9958ef0dfd90c640f7dc98df9a"
server_key_version: "encryptionkey@0"
I20260929 02:55:37.615377 20590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:37.645062 20590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:37.645921 20590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:37.646209 20590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:37.663550 20590 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:44597
I20260929 02:55:37.663589 20655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:44597 every 8 connection(s)
I20260929 02:55:37.664855 20590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:37.665264 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 20590
I20260929 02:55:37.665467 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:37.666253 15433 external_mini_cluster.cc:1428] Setting key 9d887fb99ee6bfef87b97586af2e6123
I20260929 02:55:37.669770 20656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:37.681632 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:37.668397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.15.18.126:44597)
0929 02:55:37.669633 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:37.669652 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:37.669721 (+ 69us) server_negotiation.cc:407] Connection header received
0929 02:55:37.671685 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:37.671719 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:37.672250 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:37.672654 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:37.673286 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:37.674764 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:37.676774 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:37.677514 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:37.677680 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:37.679041 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:37.679049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:37.679066 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:37.679748 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:37.679842 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:37.679870 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:37.679971 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:37.680206 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 02:55:37.680573 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":401,"threads_started":1}
I20260929 02:55:37.682358 20656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08: Bootstrap starting.
I20260929 02:55:37.685701 20656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:37.685748 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:55:37.687052 20656 log.cc:824] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:37.690037 20656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08: No bootstrap required, opened a new log
I20260929 02:55:37.697220 20656 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eb2d69100f49e1afa562f899f0bd08" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44597 } }
I20260929 02:55:37.697607 20656 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:37.697683 20656 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3eb2d69100f49e1afa562f899f0bd08, State: Initialized, Role: FOLLOWER
I20260929 02:55:37.698216 20656 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eb2d69100f49e1afa562f899f0bd08" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44597 } }
I20260929 02:55:37.698428 20656 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:37.698521 20656 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:37.698652 20656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:37.700976 20656 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eb2d69100f49e1afa562f899f0bd08" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44597 } }
I20260929 02:55:37.701486 20656 leader_election.cc:302] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3eb2d69100f49e1afa562f899f0bd08; no voters:
I20260929 02:55:37.702190 20656 leader_election.cc:288] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:37.702318 20661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:37.703070 20661 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [term 1 LEADER]: Becoming Leader. State: Replica: c3eb2d69100f49e1afa562f899f0bd08, State: Running, Role: LEADER
I20260929 02:55:37.703733 20661 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eb2d69100f49e1afa562f899f0bd08" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44597 } }
I20260929 02:55:37.704603 20656 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:37.706664 20663 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3eb2d69100f49e1afa562f899f0bd08. Latest consensus state: current_term: 1 leader_uuid: "c3eb2d69100f49e1afa562f899f0bd08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eb2d69100f49e1afa562f899f0bd08" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44597 } } }
I20260929 02:55:37.707316 20663 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:37.707309 20662 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3eb2d69100f49e1afa562f899f0bd08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eb2d69100f49e1afa562f899f0bd08" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44597 } } }
I20260929 02:55:37.708268 20662 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:37.708423 20670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:37.714253 20670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:37.723224 20670 catalog_manager.cc:1347] Generated new cluster ID: 36f26c696dee4c8fab56ca5d4c4236a0
I20260929 02:55:37.723371 20670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:37.751648 20670 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:37.752976 20670 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:37.759600 20670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3eb2d69100f49e1afa562f899f0bd08: Generated new TSK 0
I20260929 02:55:37.760445 20670 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 02:55:37.812263 20686 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 02:55:37.812460 20686 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:55:37.813197 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:37.784335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.15.18.126:44597)
0929 02:55:37.784508 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:37.784513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:37.785164 (+ 651us) server_negotiation.cc:407] Connection header received
0929 02:55:37.785479 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:37.785483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:37.785556 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:37.785673 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:55:37.786282 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:37.787034 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:37.789141 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:37.789598 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:37.789691 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:37.790324 (+ 633us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:55:37.812648 (+ 22324us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 02:55:37.812960 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52570: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 02:55:37.813320 20659 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52570: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 02:55:37.815171 20685 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:55:37.784345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44597 (local address 127.0.0.1:52570)
0929 02:55:37.785052 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:37.785072 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:37.785262 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:37.785721 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:37.785727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:37.785740 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:55:37.786132 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:37.786144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:37.787176 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:37.787182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:37.788990 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:37.789006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:37.789768 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:37.789775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:37.789959 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:37.790038 (+ 79us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:55:37.812940 (+ 22902us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 02:55:37.813332 (+ 392us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.18.126:44597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":328,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.18.126:44597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 02:55:37.817198 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 20590
I20260929 02:55:37.818459 20651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:37.911720 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 20590
2026-09-29T02:55:37Z chronyd exiting
I20260929 02:55:37.958231 15433 test_util.cc:175] -----------------------------------------------
I20260929 02:55:37.958387 15433 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790650496826590-15433-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (765 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 02:55:37.962013 15433 mini_oidc.cc:152] Starting JWKS server
I20260929 02:55:37.963795 15433 webserver.cc:460] Webserver started at https://127.0.0.1:37517/ using document root <none> and password file <none>
I20260929 02:55:37.963985 15433 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:55:37.964624 15433 webserver.cc:460] Webserver started at http://127.0.0.1:36831/ using document root <none> and password file <none>
2026-09-29T02:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:37Z Disabled control of system clock
I20260929 02:55:37.986521 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:44095
--webserver_interface=127.15.18.126
--webserver_port=0
--builtin_ntp_servers=127.15.18.84:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:44095
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37517/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/testchainca.cert with env {}
W20260929 02:55:38.298748 20693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:38.299052 20693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:38.299170 20693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:38.306386 20693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:38.306493 20693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:38.306535 20693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:38.306609 20693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:38.306658 20693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:55:38.306730 20693 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:55:38.306773 20693 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37517/jwks.json
I20260929 02:55:38.316689 20693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:44095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:44095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37517/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:38.317952 20693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:38.319207 20693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:38.330363 20698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:38.330519 20699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:38.330508 20701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:38.331557 20693 server_base.cc:1034] running on GCE node
I20260929 02:55:38.332347 20693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:38.333783 20693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:38.335003 20693 hybrid_clock.cc:648] HybridClock initialized: now 1790650538334954 us; error 38 us; skew 500 ppm
I20260929 02:55:38.335336 20693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:55:38.337705 20693 webserver.cc:460] Webserver started at http://127.15.18.126:36211/ using document root <none> and password file <none>
I20260929 02:55:38.338395 20693 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:38.338500 20693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:38.338809 20693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:38.342998 20693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6"
format_stamp: "Formatted at 2026-09-29 02:55:38 on dist-test-slave-q5f0"
server_key: "420c63f101bfdba94037c6c870e35711"
server_key_iv: "5261d2286918d1089a33460f736d6998"
server_key_version: "encryptionkey@0"
I20260929 02:55:38.343870 20693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6"
format_stamp: "Formatted at 2026-09-29 02:55:38 on dist-test-slave-q5f0"
server_key: "420c63f101bfdba94037c6c870e35711"
server_key_iv: "5261d2286918d1089a33460f736d6998"
server_key_version: "encryptionkey@0"
I20260929 02:55:38.349184 20693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 02:55:38.352783 20707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:38.354166 20693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 02:55:38.354344 20693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6"
format_stamp: "Formatted at 2026-09-29 02:55:38 on dist-test-slave-q5f0"
server_key: "420c63f101bfdba94037c6c870e35711"
server_key_iv: "5261d2286918d1089a33460f736d6998"
server_key_version: "encryptionkey@0"
I20260929 02:55:38.354560 20693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:38.378134 20693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:38.378969 20693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:38.379256 20693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:38.396428 20693 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:44095
I20260929 02:55:38.396448 20758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:44095 every 8 connection(s)
I20260929 02:55:38.397722 20693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:38.401386 20759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:38.403142 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 20693
I20260929 02:55:38.403308 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:38.403961 15433 external_mini_cluster.cc:1428] Setting key 682649db2b95f1836a1dece25ac97d3b
I20260929 02:55:38.413035 20759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6: Bootstrap starting.
I20260929 02:55:38.416500 20759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:38.417541 20759 log.cc:824] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:38.417596 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:38.405877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.15.18.126:44095)
0929 02:55:38.407048 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:38.407060 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:38.407163 (+ 103us) server_negotiation.cc:407] Connection header received
0929 02:55:38.408931 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:38.408952 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:38.409426 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:38.409775 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:38.410315 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:38.411669 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:38.413398 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:38.414061 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:38.414186 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:38.415347 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:38.415355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:38.415384 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:38.415947 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:38.416039 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:38.416059 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:38.416139 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:38.416347 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 02:55:38.416679 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":409,"threads_started":1}
I20260929 02:55:38.420329 20759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6: No bootstrap required, opened a new log
I20260929 02:55:38.421084 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:55:38.427660 20759 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44095 } }
I20260929 02:55:38.428014 20759 raft_consensus.cc:374] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:38.428097 20759 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03ca71d231a4cd28ac09b3ca3a7f7b6, State: Initialized, Role: FOLLOWER
I20260929 02:55:38.428607 20759 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44095 } }
I20260929 02:55:38.428817 20759 raft_consensus.cc:388] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:38.428938 20759 raft_consensus.cc:482] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:38.429081 20759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:38.431363 20759 raft_consensus.cc:504] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44095 } }
I20260929 02:55:38.431856 20759 leader_election.cc:302] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c03ca71d231a4cd28ac09b3ca3a7f7b6; no voters:
I20260929 02:55:38.432376 20759 leader_election.cc:288] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:38.432504 20764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:38.433120 20764 raft_consensus.cc:686] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [term 1 LEADER]: Becoming Leader. State: Replica: c03ca71d231a4cd28ac09b3ca3a7f7b6, State: Running, Role: LEADER
I20260929 02:55:38.433741 20764 consensus_queue.cc:237] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44095 } }
I20260929 02:55:38.434402 20759 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:38.436487 20766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03ca71d231a4cd28ac09b3ca3a7f7b6. Latest consensus state: current_term: 1 leader_uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44095 } } }
I20260929 02:55:38.436973 20766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:38.436820 20765 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03ca71d231a4cd28ac09b3ca3a7f7b6" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44095 } } }
I20260929 02:55:38.437134 20765 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:38.438508 20771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:38.444631 20771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:38.453670 20771 catalog_manager.cc:1347] Generated new cluster ID: 3ed4115437b643f78cff855cc62f5011
I20260929 02:55:38.453783 20771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:38.473961 20771 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:38.475214 20771 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:38.482016 20771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c03ca71d231a4cd28ac09b3ca3a7f7b6: Generated new TSK 0
I20260929 02:55:38.482846 20771 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 02:55:38.516814 20789 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 02:55:38.516934 20789 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:55:38.518174 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:38.497761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.15.18.126:44095)
0929 02:55:38.497915 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:38.497919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:38.498309 (+ 390us) server_negotiation.cc:407] Connection header received
0929 02:55:38.498499 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:38.498505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:38.498593 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:38.498766 (+ 173us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:55:38.499216 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:38.500378 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:38.501961 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:38.502661 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:38.502792 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:38.503305 (+ 513us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:55:38.517463 (+ 14158us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 02:55:38.517884 (+ 421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 02:55:38.518283 20788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:55:38.497612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44095 (local address 127.0.0.1:48804)
0929 02:55:38.498205 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:38.498224 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:38.498348 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:38.498766 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:38.498769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:38.498778 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:55:38.499049 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:38.499058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:38.500549 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:38.500557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:38.501714 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:38.501727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:38.502862 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:38.502867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:38.503005 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:38.503043 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:55:38.517716 (+ 14673us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 02:55:38.517976 (+ 260us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.18.126:44095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":313,"threads_started":1}
W20260929 02:55:38.518298 20762 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 02:55:38.519304 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 20693
I20260929 02:55:38.520510 20754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:38.612105 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 20693
2026-09-29T02:55:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (701 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 02:55:38.663331 15433 mini_oidc.cc:152] Starting JWKS server
I20260929 02:55:38.665179 15433 webserver.cc:460] Webserver started at https://127.0.0.1:42509/ using document root <none> and password file <none>
I20260929 02:55:38.665406 15433 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:55:38.665985 15433 webserver.cc:460] Webserver started at http://127.0.0.1:43647/ using document root <none> and password file <none>
2026-09-29T02:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:38Z Disabled control of system clock
I20260929 02:55:38.690304 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:41921
--webserver_interface=127.15.18.126
--webserver_port=0
--builtin_ntp_servers=127.15.18.84:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:41921
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42509/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 02:55:39.007611 20796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:39.007930 20796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:39.008028 20796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:39.015754 20796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:39.015880 20796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:39.015916 20796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:39.015970 20796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:39.016026 20796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:55:39.016109 20796 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:55:39.016166 20796 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42509/jwks.json
I20260929 02:55:39.025921 20796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:41921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:41921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42509/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:39.027259 20796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:39.028990 20796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:39.040387 20801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:39.040359 20804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:39.040329 20802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:39.040900 20796 server_base.cc:1034] running on GCE node
I20260929 02:55:39.041636 20796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:39.043009 20796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:39.044191 20796 hybrid_clock.cc:648] HybridClock initialized: now 1790650539044149 us; error 41 us; skew 500 ppm
I20260929 02:55:39.044513 20796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:55:39.046754 20796 webserver.cc:460] Webserver started at http://127.15.18.126:38439/ using document root <none> and password file <none>
I20260929 02:55:39.047415 20796 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:39.047498 20796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:39.047819 20796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:39.050695 20796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "0d43c9d88ef14d16ad35e4d29efa724a"
format_stamp: "Formatted at 2026-09-29 02:55:39 on dist-test-slave-q5f0"
server_key: "9482991d44d73328a5c0d94b269bdfd8"
server_key_iv: "8be6763a94f56ba36cb0effea03dc0b5"
server_key_version: "encryptionkey@0"
I20260929 02:55:39.051496 20796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "0d43c9d88ef14d16ad35e4d29efa724a"
format_stamp: "Formatted at 2026-09-29 02:55:39 on dist-test-slave-q5f0"
server_key: "9482991d44d73328a5c0d94b269bdfd8"
server_key_iv: "8be6763a94f56ba36cb0effea03dc0b5"
server_key_version: "encryptionkey@0"
I20260929 02:55:39.056839 20796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 02:55:39.060351 20810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:39.061726 20796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:39.061885 20796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "0d43c9d88ef14d16ad35e4d29efa724a"
format_stamp: "Formatted at 2026-09-29 02:55:39 on dist-test-slave-q5f0"
server_key: "9482991d44d73328a5c0d94b269bdfd8"
server_key_iv: "8be6763a94f56ba36cb0effea03dc0b5"
server_key_version: "encryptionkey@0"
I20260929 02:55:39.062114 20796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:39.095793 20796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:39.096658 20796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:39.096969 20796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:39.112928 20796 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:41921
I20260929 02:55:39.112952 20861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:41921 every 8 connection(s)
I20260929 02:55:39.114209 20796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:39.117898 20862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:39.118265 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 20796
I20260929 02:55:39.118448 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:39.119040 15433 external_mini_cluster.cc:1428] Setting key bea8b3376efd19028feaf3610cb1f5f2
I20260929 02:55:39.129738 20862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a: Bootstrap starting.
I20260929 02:55:39.132194 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:39.120832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.15.18.126:41921)
0929 02:55:39.121868 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:39.121880 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:39.121922 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:55:39.123764 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:39.123787 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:39.124277 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:39.124634 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:39.125159 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.126424 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.127743 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.128453 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.128629 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:39.129771 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:39.129779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:39.129794 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:39.130390 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:39.130469 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:39.130491 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:39.130603 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:39.130817 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 02:55:39.131141 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":348,"threads_started":1}
I20260929 02:55:39.133183 20862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:39.134270 20862 log.cc:824] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:39.136281 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:55:39.137480 20862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a: No bootstrap required, opened a new log
I20260929 02:55:39.144775 20862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d43c9d88ef14d16ad35e4d29efa724a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41921 } }
I20260929 02:55:39.145136 20862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:39.145211 20862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d43c9d88ef14d16ad35e4d29efa724a, State: Initialized, Role: FOLLOWER
I20260929 02:55:39.145707 20862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d43c9d88ef14d16ad35e4d29efa724a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41921 } }
I20260929 02:55:39.145933 20862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:39.146060 20862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:39.146207 20862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:39.148317 20862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d43c9d88ef14d16ad35e4d29efa724a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41921 } }
I20260929 02:55:39.148777 20862 leader_election.cc:302] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d43c9d88ef14d16ad35e4d29efa724a; no voters:
I20260929 02:55:39.149328 20862 leader_election.cc:288] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:39.149459 20867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:39.150099 20867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [term 1 LEADER]: Becoming Leader. State: Replica: 0d43c9d88ef14d16ad35e4d29efa724a, State: Running, Role: LEADER
I20260929 02:55:39.150666 20867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d43c9d88ef14d16ad35e4d29efa724a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41921 } }
I20260929 02:55:39.151257 20862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:39.153232 20869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d43c9d88ef14d16ad35e4d29efa724a. Latest consensus state: current_term: 1 leader_uuid: "0d43c9d88ef14d16ad35e4d29efa724a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d43c9d88ef14d16ad35e4d29efa724a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41921 } } }
I20260929 02:55:39.153514 20868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d43c9d88ef14d16ad35e4d29efa724a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d43c9d88ef14d16ad35e4d29efa724a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41921 } } }
I20260929 02:55:39.153708 20869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:39.153770 20868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:39.156729 20874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:39.166349 20874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:39.176595 20874 catalog_manager.cc:1347] Generated new cluster ID: fc3b276d1edf4371b7415aaf8427caf5
I20260929 02:55:39.176725 20874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:39.193048 20874 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:39.194641 20874 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:39.205870 20874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d43c9d88ef14d16ad35e4d29efa724a: Generated new TSK 0
I20260929 02:55:39.207104 20874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:55:39.238772 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:39.230233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.15.18.126:41921)
0929 02:55:39.230428 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:39.230434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:39.230740 (+ 306us) server_negotiation.cc:407] Connection header received
0929 02:55:39.231033 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:39.231037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:39.231107 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:39.231247 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:55:39.231797 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.232540 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.234526 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.235252 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.235357 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:39.235991 (+ 634us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:55:39.238134 (+ 2143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42509/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 02:55:39.238474 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42509/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 02:55:39.238911 20865 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42509/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 02:55:39.241063 20891 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:55:39.230038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:41921 (local address 127.0.0.1:56904)
0929 02:55:39.230630 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:39.230648 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:39.230807 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:39.231274 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:39.231279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:39.231293 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:55:39.231647 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:39.231660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.232664 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.232670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:39.234369 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:39.234384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.235446 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.235453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:39.235670 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:39.235718 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:55:39.238366 (+ 2648us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42509/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 02:55:39.240555 (+ 2189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.18.126:41921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42509/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":1859,"thread_start_us":306,"threads_started":1}
I20260929 02:55:39.242270 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 20796
I20260929 02:55:39.243585 20857 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 02:55:39.250350 20892 webserver.cc:533] Webserver: sslize failed:
E20260929 02:55:39.250461 20892 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:55:39.335562 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 20796
2026-09-29T02:55:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T02:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:39Z Disabled control of system clock
I20260929 02:55:39.412714 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43721
--webserver_interface=127.15.18.126
--webserver_port=0
--builtin_ntp_servers=127.15.18.84:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43721
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/testchainca.cert with env {}
W20260929 02:55:39.724982 20897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:39.725282 20897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:39.725410 20897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:39.732597 20897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:39.732698 20897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:39.732741 20897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:39.732821 20897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:39.732877 20897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:55:39.732955 20897 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:55:39.732999 20897 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 02:55:39.742755 20897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:39.744068 20897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:39.745321 20897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:39.756871 20903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:39.756850 20905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:39.756820 20902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:39.757705 20897 server_base.cc:1034] running on GCE node
I20260929 02:55:39.758491 20897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:39.760053 20897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:39.761284 20897 hybrid_clock.cc:648] HybridClock initialized: now 1790650539761227 us; error 38 us; skew 500 ppm
I20260929 02:55:39.761631 20897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:55:39.763952 20897 webserver.cc:460] Webserver started at http://127.15.18.126:40919/ using document root <none> and password file <none>
I20260929 02:55:39.764659 20897 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:39.764761 20897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:39.765045 20897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:39.768033 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "b61cd35c93734bd6b04c91e986216070"
format_stamp: "Formatted at 2026-09-29 02:55:39 on dist-test-slave-q5f0"
server_key: "4359fd43efe1422384379c5831f1bac8"
server_key_iv: "6a58dd346fca18a75e927176e1401e24"
server_key_version: "encryptionkey@0"
I20260929 02:55:39.768819 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "b61cd35c93734bd6b04c91e986216070"
format_stamp: "Formatted at 2026-09-29 02:55:39 on dist-test-slave-q5f0"
server_key: "4359fd43efe1422384379c5831f1bac8"
server_key_iv: "6a58dd346fca18a75e927176e1401e24"
server_key_version: "encryptionkey@0"
I20260929 02:55:39.774068 20897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:55:39.777648 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:39.779168 20897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:55:39.779356 20897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "b61cd35c93734bd6b04c91e986216070"
format_stamp: "Formatted at 2026-09-29 02:55:39 on dist-test-slave-q5f0"
server_key: "4359fd43efe1422384379c5831f1bac8"
server_key_iv: "6a58dd346fca18a75e927176e1401e24"
server_key_version: "encryptionkey@0"
I20260929 02:55:39.779640 20897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:39.808770 20897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:39.809653 20897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:39.809926 20897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:39.827078 20897 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:43721
I20260929 02:55:39.827111 20962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:43721 every 8 connection(s)
I20260929 02:55:39.828415 20897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:39.831207 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 20897
I20260929 02:55:39.831384 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:39.832115 15433 external_mini_cluster.cc:1428] Setting key 6973d769c5cb6809ae1db6721bdb90e2
I20260929 02:55:39.832315 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:39.846285 20963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070: Bootstrap starting.
I20260929 02:55:39.846758 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:39.833978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.15.18.126:43721)
0929 02:55:39.834973 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:39.834986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:39.835034 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:55:39.837040 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:39.837067 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:39.837648 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:39.838054 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:39.838722 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.840116 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.841859 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.842570 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.842728 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:39.844106 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:39.844119 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:39.844136 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:55:39.844778 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:39.844869 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:55:39.844895 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:39.845012 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:39.845250 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 02:55:39.845622 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":349,"threads_started":1}
I20260929 02:55:39.849956 20963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:39.851127 20963 log.cc:824] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:39.851249 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:55:39.854489 20963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070: No bootstrap required, opened a new log
I20260929 02:55:39.861935 20963 raft_consensus.cc:348] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61cd35c93734bd6b04c91e986216070" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43721 } }
I20260929 02:55:39.862434 20963 raft_consensus.cc:374] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:39.862541 20963 raft_consensus.cc:729] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b61cd35c93734bd6b04c91e986216070, State: Initialized, Role: FOLLOWER
I20260929 02:55:39.863149 20963 consensus_queue.cc:260] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61cd35c93734bd6b04c91e986216070" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43721 } }
I20260929 02:55:39.863384 20963 raft_consensus.cc:388] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:39.863538 20963 raft_consensus.cc:482] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:39.863703 20963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:39.865847 20963 raft_consensus.cc:504] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61cd35c93734bd6b04c91e986216070" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43721 } }
I20260929 02:55:39.866377 20963 leader_election.cc:302] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b61cd35c93734bd6b04c91e986216070; no voters:
I20260929 02:55:39.867007 20963 leader_election.cc:288] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:39.867182 20968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:39.867888 20968 raft_consensus.cc:686] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [term 1 LEADER]: Becoming Leader. State: Replica: b61cd35c93734bd6b04c91e986216070, State: Running, Role: LEADER
I20260929 02:55:39.868396 20968 consensus_queue.cc:237] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61cd35c93734bd6b04c91e986216070" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43721 } }
I20260929 02:55:39.869174 20963 sys_catalog.cc:564] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:39.871017 20970 sys_catalog.cc:455] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b61cd35c93734bd6b04c91e986216070. Latest consensus state: current_term: 1 leader_uuid: "b61cd35c93734bd6b04c91e986216070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61cd35c93734bd6b04c91e986216070" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43721 } } }
I20260929 02:55:39.871027 20969 sys_catalog.cc:455] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b61cd35c93734bd6b04c91e986216070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61cd35c93734bd6b04c91e986216070" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43721 } } }
I20260929 02:55:39.871495 20969 sys_catalog.cc:458] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:39.871501 20970 sys_catalog.cc:458] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:39.876641 20973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:39.881830 20973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:39.890661 20973 catalog_manager.cc:1347] Generated new cluster ID: fd246b3b5613424ca4d1c1272b2239c7
I20260929 02:55:39.890799 20973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:39.904335 20973 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:39.906139 20973 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:39.925050 20973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b61cd35c93734bd6b04c91e986216070: Generated new TSK 0
I20260929 02:55:39.926192 20973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:55:39.993194 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:39.938377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.15.18.126:43721)
0929 02:55:39.938521 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:39.938526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:39.938974 (+ 448us) server_negotiation.cc:407] Connection header received
0929 02:55:39.939202 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:39.939206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:39.939286 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:39.939407 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:55:39.939885 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.940581 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.941937 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.942385 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.942489 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:39.942951 (+ 462us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:55:39.992457 (+ 49506us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 02:55:39.992881 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38134: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 02:55:39.993348 20966 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38134: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 02:55:39.993427 20992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:55:39.938379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43721 (local address 127.0.0.1:38134)
0929 02:55:39.938872 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:39.938888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:39.939026 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:39.939443 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:39.939447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:39.939460 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:55:39.939738 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:39.939747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.940715 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.940720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:39.941807 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:39.941817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:39.942548 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:39.942554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:39.942681 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:39.942713 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:55:39.992755 (+ 50042us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 02:55:39.993033 (+ 278us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.18.126:43721: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":260,"threads_started":1}
I20260929 02:55:39.995110 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 20897
I20260929 02:55:39.996397 20958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:40.094450 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 20897
2026-09-29T02:55:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (737 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (400 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (397 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:40 dist-test-slave-q5f0 krb5kdc[21007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:40 dist-test-slave-q5f0 krb5kdc[21007](info): set up 2 sockets
Sep 29 02:55:40 dist-test-slave-q5f0 krb5kdc[21007](info): commencing operation
krb5kdc: starting...
W20260929 02:55:41.896220 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:55:42 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:55:42.095868 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43255
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:42.412618 21020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:42.412925 21020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:42.413034 21020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:42.420269 21020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:42.420377 21020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:42.420432 21020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:42.420511 21020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:42.420569 21020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:42.430357 21020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:42.431727 21020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:42.433012 21020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:42.444387 21025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:42.444455 21026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:42.444422 21028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:42.445555 21020 server_base.cc:1034] running on GCE node
I20260929 02:55:42.446349 21020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:42.447830 21020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:42.449019 21020 hybrid_clock.cc:648] HybridClock initialized: now 1790650542448961 us; error 53 us; skew 500 ppm
I20260929 02:55:42.449365 21020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:42 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650542, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:42.453441 21020 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:55:42.454974 21020 webserver.cc:460] Webserver started at http://127.15.18.126:43531/ using document root <none> and password file <none>
I20260929 02:55:42.455711 21020 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:42.455826 21020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:42.456120 21020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:42.459090 21020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "eb614dbd37114d318305f46ad77045ab"
format_stamp: "Formatted at 2026-09-29 02:55:42 on dist-test-slave-q5f0"
server_key: "ed5976a636eb0e36156e7293d1789099"
server_key_iv: "668c4239508bc832afedce5f6695f0e8"
server_key_version: "encryptionkey@0"
I20260929 02:55:42.459883 21020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "eb614dbd37114d318305f46ad77045ab"
format_stamp: "Formatted at 2026-09-29 02:55:42 on dist-test-slave-q5f0"
server_key: "ed5976a636eb0e36156e7293d1789099"
server_key_iv: "668c4239508bc832afedce5f6695f0e8"
server_key_version: "encryptionkey@0"
I20260929 02:55:42.464982 21020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:55:42.468449 21035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:42.469808 21020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:42.469962 21020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "eb614dbd37114d318305f46ad77045ab"
format_stamp: "Formatted at 2026-09-29 02:55:42 on dist-test-slave-q5f0"
server_key: "ed5976a636eb0e36156e7293d1789099"
server_key_iv: "668c4239508bc832afedce5f6695f0e8"
server_key_version: "encryptionkey@0"
I20260929 02:55:42.470137 21020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:42.490839 21020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:42.495143 21020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:42.495436 21020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:42.514221 21020 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:43255
I20260929 02:55:42.514227 21086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:43255 every 8 connection(s)
I20260929 02:55:42.515652 21020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:42.519322 21087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:42.524950 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 21020
I20260929 02:55:42.525115 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:42.525725 15433 external_mini_cluster.cc:1428] Setting key c7735c8c1cc1241c3f4458b9fb52bab3
I20260929 02:55:42.530720 21087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab: Bootstrap starting.
I20260929 02:55:42.534370 21087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:42.535485 21087 log.cc:824] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:42.538656 21087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab: No bootstrap required, opened a new log
Sep 29 02:55:42 dist-test-slave-q5f0 krb5kdc[21007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:42.548115 21087 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb614dbd37114d318305f46ad77045ab" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43255 } }
I20260929 02:55:42.548609 21087 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:42.548736 21087 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb614dbd37114d318305f46ad77045ab, State: Initialized, Role: FOLLOWER
I20260929 02:55:42.549408 21087 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb614dbd37114d318305f46ad77045ab" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43255 } }
I20260929 02:55:42.549715 21087 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:42.549855 21087 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:42.550024 21087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:42.550172 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:42.527761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.15.18.126:43255)
0929 02:55:42.528559 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:42.528574 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:42.528622 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:55:42.530866 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:42.530893 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:42.531406 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:42.531848 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:42.533131 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:42.534988 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:42.536613 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:42.537552 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:42.537733 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:42.541710 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:42.541721 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:42.541741 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:42.541788 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:42.545692 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:42.546387 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:42.546396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:42.546406 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:42.546624 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:42.546961 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:42.546966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:42.546970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:42.547702 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:42.547968 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:42.548348 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 02:55:42.548782 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":352,"threads_started":1}
I20260929 02:55:42.552392 21087 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb614dbd37114d318305f46ad77045ab" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43255 } }
I20260929 02:55:42.552934 21087 leader_election.cc:302] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb614dbd37114d318305f46ad77045ab; no voters:
I20260929 02:55:42.553503 21087 leader_election.cc:288] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:42.553617 21092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:42.554646 21092 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [term 1 LEADER]: Becoming Leader. State: Replica: eb614dbd37114d318305f46ad77045ab, State: Running, Role: LEADER
I20260929 02:55:42.555265 21092 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb614dbd37114d318305f46ad77045ab" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43255 } }
I20260929 02:55:42.555595 21087 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:42.558043 21094 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb614dbd37114d318305f46ad77045ab. Latest consensus state: current_term: 1 leader_uuid: "eb614dbd37114d318305f46ad77045ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb614dbd37114d318305f46ad77045ab" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43255 } } }
I20260929 02:55:42.558400 21094 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:42.558928 21093 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb614dbd37114d318305f46ad77045ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb614dbd37114d318305f46ad77045ab" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43255 } } }
I20260929 02:55:42.559229 21093 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:42.559294 21100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:42.565053 21100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:42.574214 21100 catalog_manager.cc:1347] Generated new cluster ID: de6e0a62075e4ee4bde0d03953f6dc5e
I20260929 02:55:42.574353 21100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:42.588167 21100 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:42.590032 21100 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:42.605197 21100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb614dbd37114d318305f46ad77045ab: Generated new TSK 0
I20260929 02:55:42.606388 21100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:55:42.655313 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43255
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:42.977393 21114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:42.977676 21114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:42.977792 21114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:42.985008 21114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:42.985118 21114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:42.985257 21114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:42.994997 21114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:42.996325 21114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:42.997816 21114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:43.010443 21122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:43.010512 21119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:43.010486 21120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:43.011339 21114 server_base.cc:1034] running on GCE node
I20260929 02:55:43.011988 21114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:43.012939 21114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:43.014120 21114 hybrid_clock.cc:648] HybridClock initialized: now 1790650543014068 us; error 51 us; skew 500 ppm
I20260929 02:55:43.014389 21114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:43 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650543, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:43.017621 21114 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:55:43.019150 21114 webserver.cc:460] Webserver started at http://127.15.18.65:33957/ using document root <none> and password file <none>
I20260929 02:55:43.019892 21114 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:43.019996 21114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:43.020290 21114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:43.023298 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "28e6367204f74c689643cbe758e9c78e"
format_stamp: "Formatted at 2026-09-29 02:55:43 on dist-test-slave-q5f0"
server_key: "27fd089679d14f8a43d036d37212ba1d"
server_key_iv: "06cd2faaeffec6ef23b1f01b226a1b58"
server_key_version: "encryptionkey@0"
I20260929 02:55:43.024143 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "28e6367204f74c689643cbe758e9c78e"
format_stamp: "Formatted at 2026-09-29 02:55:43 on dist-test-slave-q5f0"
server_key: "27fd089679d14f8a43d036d37212ba1d"
server_key_iv: "06cd2faaeffec6ef23b1f01b226a1b58"
server_key_version: "encryptionkey@0"
I20260929 02:55:43.029313 21114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:55:43.032908 21129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:43.034252 21114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:43.034400 21114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "28e6367204f74c689643cbe758e9c78e"
format_stamp: "Formatted at 2026-09-29 02:55:43 on dist-test-slave-q5f0"
server_key: "27fd089679d14f8a43d036d37212ba1d"
server_key_iv: "06cd2faaeffec6ef23b1f01b226a1b58"
server_key_version: "encryptionkey@0"
I20260929 02:55:43.034585 21114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:43.047819 21114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:43.052204 21114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:43.052459 21114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:43.053431 21114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:43.054745 21114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:43.054823 21114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:43.054935 21114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:43.054996 21114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:43.086740 21114 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:35607
I20260929 02:55:43.086787 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:35607 every 8 connection(s)
I20260929 02:55:43.088024 21114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:43.094281 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 21114
I20260929 02:55:43.094461 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:43.095103 15433 external_mini_cluster.cc:1428] Setting key 0dd722bc53fb65a069fa1cf958389037
Sep 29 02:55:43 dist-test-slave-q5f0 krb5kdc[21007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650543, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:43.108213 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:43.090471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:53931 (local address 127.15.18.126:43255)
0929 02:55:43.090876 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:43.090881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:43.091885 (+ 1004us) server_negotiation.cc:407] Connection header received
0929 02:55:43.093660 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:43.093666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:43.093747 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:43.093850 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:43.095474 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:43.096352 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:43.098064 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:43.098493 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:43.098605 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:43.102311 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:43.102317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:43.102320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:43.102356 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:43.104948 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:43.105814 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:43.105821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:43.105823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:43.105948 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:43.106383 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:43.106388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:43.106391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:43.106668 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:43.106807 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:43.107588 (+ 781us) server_negotiation.cc:299] Negotiation successful
0929 02:55:43.107820 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":191,"threads_started":1}
I20260929 02:55:43.109211 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:43.091021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43255 (local address 127.15.18.65:53931)
0929 02:55:43.091733 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:43.091770 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:43.093395 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:43.094111 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:43.094120 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:43.094686 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:43.095306 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:43.095326 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:43.096507 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:43.096511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:43.097919 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:43.097932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:43.098715 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:43.098722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:43.098945 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:43.099630 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:43.099654 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:43.102031 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:43.105112 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:43.105123 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:43.105143 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:43.105689 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:43.106057 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:43.106062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:43.106066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:43.106271 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:43.106804 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:43.106814 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:43.107182 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 02:55:43.107433 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 02:55:43.108029 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":279,"threads_started":1}
I20260929 02:55:43.110867 21242 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43255
I20260929 02:55:43.111263 21242 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:43.112064 21242 heartbeater.cc:511] Master 127.15.18.126:43255 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
I20260929 02:55:43.114207 21052 ts_manager.cc:194] Registered new tserver with Master: 28e6367204f74c689643cbe758e9c78e (127.15.18.65:35607)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
I20260929 02:55:43.115876 21052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:53931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:55:43.139067 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43255
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:43.454674 21248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:43.455068 21248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:43.455264 21248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:43.462845 21248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:43.462986 21248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:43.463147 21248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:43.473248 21248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:43.474566 21248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:43.476007 21248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:43.488768 21254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:43.488864 21256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:43.488873 21253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:43.489398 21248 server_base.cc:1034] running on GCE node
I20260929 02:55:43.490021 21248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:43.490938 21248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:43.492158 21248 hybrid_clock.cc:648] HybridClock initialized: now 1790650543492104 us; error 44 us; skew 500 ppm
I20260929 02:55:43.492471 21248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:43 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650543, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:43.495590 21248 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:55:43.497196 21248 webserver.cc:460] Webserver started at http://127.15.18.66:44783/ using document root <none> and password file <none>
I20260929 02:55:43.497851 21248 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:43.497967 21248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:43.498252 21248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:43.501262 21248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "c7639a7455fe46bab4747d46995cca08"
format_stamp: "Formatted at 2026-09-29 02:55:43 on dist-test-slave-q5f0"
server_key: "e0c87127e7d8cb58ea31579bc1ff1bd1"
server_key_iv: "17e974d5bf5f8c7815bc2299557e3ce0"
server_key_version: "encryptionkey@0"
I20260929 02:55:43.502017 21248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "c7639a7455fe46bab4747d46995cca08"
format_stamp: "Formatted at 2026-09-29 02:55:43 on dist-test-slave-q5f0"
server_key: "e0c87127e7d8cb58ea31579bc1ff1bd1"
server_key_iv: "17e974d5bf5f8c7815bc2299557e3ce0"
server_key_version: "encryptionkey@0"
I20260929 02:55:43.507092 21248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:55:43.510488 21263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:43.511906 21248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:55:43.512063 21248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "c7639a7455fe46bab4747d46995cca08"
format_stamp: "Formatted at 2026-09-29 02:55:43 on dist-test-slave-q5f0"
server_key: "e0c87127e7d8cb58ea31579bc1ff1bd1"
server_key_iv: "17e974d5bf5f8c7815bc2299557e3ce0"
server_key_version: "encryptionkey@0"
I20260929 02:55:43.512252 21248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:43.552050 21248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:43.556294 21248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:43.556555 21248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:43.557548 21248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:43.558884 21248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:43.558960 21248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:43.559082 21248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:43.559137 21248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:43.588327 21248 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:37035
I20260929 02:55:43.588402 21375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:37035 every 8 connection(s)
I20260929 02:55:43.589617 21248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:43.598405 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 21248
I20260929 02:55:43.598539 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:43.598976 15433 external_mini_cluster.cc:1428] Setting key cae25b0dcdf2e172c01b7db1ebd531fb
Sep 29 02:55:43 dist-test-slave-q5f0 krb5kdc[21007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650543, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:43.610384 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:43.592065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:37579 (local address 127.15.18.126:43255)
0929 02:55:43.592228 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:43.592233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:43.593451 (+ 1218us) server_negotiation.cc:407] Connection header received
0929 02:55:43.595230 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:43.595237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:43.595321 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:43.595408 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:43.596997 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:43.597718 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:43.599356 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:43.599911 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:43.600023 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:43.604169 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:43.604175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:43.604179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:43.604216 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:43.606840 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:43.607803 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:43.607807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:43.607810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:43.607948 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:43.608445 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:43.608449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:43.608451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:43.608782 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:43.608896 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:43.609835 (+ 939us) server_negotiation.cc:299] Negotiation successful
0929 02:55:43.610067 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:55:43.611649 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:43.592542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43255 (local address 127.15.18.66:37579)
0929 02:55:43.593278 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:43.593315 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:43.594950 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:43.595682 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:43.595691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:43.596230 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:43.596805 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:43.596824 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:43.597875 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:43.597879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:43.599214 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:43.599227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:43.600131 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:43.600138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:43.600365 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:43.601060 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:43.601096 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:43.603906 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:43.606984 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:43.606994 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:43.607015 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:43.607641 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:43.608060 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:43.608064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:43.608067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:43.608315 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:43.608899 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:43.608907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:43.609364 (+ 457us) client_negotiation.cc:769] Sending connection context
0929 02:55:43.609684 (+ 320us) client_negotiation.cc:240] Negotiation successful
0929 02:55:43.610242 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":299,"threads_started":1}
I20260929 02:55:43.613220 21376 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43255
I20260929 02:55:43.613523 21376 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:43.614266 21376 heartbeater.cc:511] Master 127.15.18.126:43255 requested a full tablet report, sending...
I20260929 02:55:43.615901 21052 ts_manager.cc:194] Registered new tserver with Master: c7639a7455fe46bab4747d46995cca08 (127.15.18.66:37035)
I20260929 02:55:43.617105 21052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:37579
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:55:43.646664 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43255
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:43.961436 21381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:43.961725 21381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:43.961861 21381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:43.969260 21381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:43.969409 21381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:43.969596 21381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:43.979420 21381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:43.980798 21381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:43.982057 21381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:43.995165 21389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:43.995251 21386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:43.995201 21387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:43.997188 21381 server_base.cc:1034] running on GCE node
I20260929 02:55:43.997787 21381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:43.998765 21381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:43.999969 21381 hybrid_clock.cc:648] HybridClock initialized: now 1790650543999905 us; error 53 us; skew 500 ppm
I20260929 02:55:44.000281 21381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:43 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650543, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:44.003328 21381 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:55:44.004889 21381 webserver.cc:460] Webserver started at http://127.15.18.67:33653/ using document root <none> and password file <none>
I20260929 02:55:44.005591 21381 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:44.005702 21381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:44.005987 21381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:44.009063 21381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "4c5dd4d61887430f8850260296f830f3"
format_stamp: "Formatted at 2026-09-29 02:55:44 on dist-test-slave-q5f0"
server_key: "c5629bd6ac9f66a784d32b2a770ee629"
server_key_iv: "050dcfc4d2cfd776d8e420acb711fef5"
server_key_version: "encryptionkey@0"
I20260929 02:55:44.009889 21381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "4c5dd4d61887430f8850260296f830f3"
format_stamp: "Formatted at 2026-09-29 02:55:44 on dist-test-slave-q5f0"
server_key: "c5629bd6ac9f66a784d32b2a770ee629"
server_key_iv: "050dcfc4d2cfd776d8e420acb711fef5"
server_key_version: "encryptionkey@0"
I20260929 02:55:44.015254 21381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:55:44.018874 21396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:44.020324 21381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:44.020493 21381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "4c5dd4d61887430f8850260296f830f3"
format_stamp: "Formatted at 2026-09-29 02:55:44 on dist-test-slave-q5f0"
server_key: "c5629bd6ac9f66a784d32b2a770ee629"
server_key_iv: "050dcfc4d2cfd776d8e420acb711fef5"
server_key_version: "encryptionkey@0"
I20260929 02:55:44.020690 21381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:44.054755 21381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:44.059021 21381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:44.059280 21381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:44.060247 21381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:44.061525 21381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:44.061599 21381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:44.061719 21381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:44.061769 21381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:44.093667 21381 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:41333
I20260929 02:55:44.093732 21508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:41333 every 8 connection(s)
I20260929 02:55:44.094842 21381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:44.095496 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 21381
I20260929 02:55:44.095685 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:44.096225 15433 external_mini_cluster.cc:1428] Setting key ef48b1fc86b54c8daef901005d24cc03
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650543, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:55:44.114996 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:44.097388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:56679 (local address 127.15.18.126:43255)
0929 02:55:44.097570 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:44.097576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:44.099090 (+ 1514us) server_negotiation.cc:407] Connection header received
0929 02:55:44.100925 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:44.100929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:44.100998 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:44.101083 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:44.102511 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.103239 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.104842 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.105235 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.105331 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:44.108924 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:44.108930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:44.108933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:44.108965 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:44.112144 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.112821 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.112825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.112828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.112961 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.113387 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.113391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.113393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.113656 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:44.113795 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:44.114451 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 02:55:44.114656 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:55:44.115978 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:44.098071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43255 (local address 127.15.18.67:56679)
0929 02:55:44.098931 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:44.098979 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:44.100658 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:44.101256 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:44.101263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:44.101782 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:44.102354 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:44.102370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.103374 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.103377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:44.104714 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:44.104726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.105477 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.105483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:44.105706 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:44.106424 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:44.106456 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:44.108714 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:44.112275 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:44.112285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:44.112303 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:44.112699 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:44.113052 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:44.113056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:44.113078 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:44.113275 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:44.113773 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:44.113781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:44.114101 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 02:55:44.114334 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 02:55:44.114848 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":346,"threads_started":1}
I20260929 02:55:44.117322 21509 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43255
I20260929 02:55:44.117602 21509 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:44.118289 21509 heartbeater.cc:511] Master 127.15.18.126:43255 requested a full tablet report, sending...
I20260929 02:55:44.119261 21242 heartbeater.cc:503] Master 127.15.18.126:43255 was elected leader, sending a full tablet report...
I20260929 02:55:44.119704 21052 ts_manager.cc:194] Registered new tserver with Master: 4c5dd4d61887430f8850260296f830f3 (127.15.18.67:41333)
I20260929 02:55:44.120641 21052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:56679
I20260929 02:55:44.123759 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/test-admin.keytab.
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:44.148758 15433 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 02:55:44.153779 21516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:55:44.151322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.15.18.65:35607)
0929 02:55:44.151987 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:44.151994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:44.152021 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:55:44.152079 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:44.152086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:44.152307 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:44.152434 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:44.153360 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35688: BlockingRecv error: recv got EOF from 127.0.0.1:35688 (error 108)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":338,"threads_started":1}
W20260929 02:55:44.154062 15433 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5cc)
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:44.155658 15433 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:44.157120 15433 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:44.171924 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:44.158702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.15.18.65:35607)
0929 02:55:44.158919 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:44.158925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:44.159238 (+ 313us) server_negotiation.cc:407] Connection header received
0929 02:55:44.159437 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:44.159441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:44.159504 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:44.159618 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:44.160605 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.161593 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.162785 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.163392 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.163492 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:44.166783 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:44.166793 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:44.166807 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:44.166839 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:44.169577 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.170184 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.170191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.170193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.170304 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.170578 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.170581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.170583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.170916 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:44.171124 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:44.171381 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 02:55:44.171628 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 02:55:44.177260 21516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:55:44.175729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.15.18.65:35607)
0929 02:55:44.175924 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:44.175929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:44.176087 (+ 158us) server_negotiation.cc:407] Connection header received
0929 02:55:44.176197 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:44.176200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:44.176263 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:44.176336 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:44.177073 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35708: BlockingRecv error: recv got EOF from 127.0.0.1:35708 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 02:55:44.177425 15433 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5cc' of type FILE
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:44.178781 15433 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:44.180003 15433 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:44.193444 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:44.181167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.15.18.65:35607)
0929 02:55:44.181344 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:44.181349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:44.181533 (+ 184us) server_negotiation.cc:407] Connection header received
0929 02:55:44.181666 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:44.181670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:44.181724 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:44.181791 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:44.182682 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.183376 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.184600 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.185050 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.185154 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:44.188776 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:44.188782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:44.188785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:44.188821 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:44.191219 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.191877 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.191881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.191883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.191994 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.192299 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.192302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.192304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.192557 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:44.192733 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:44.192987 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 02:55:44.193141 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 02:55:44.197634 21516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:55:44.196096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.15.18.65:35607)
0929 02:55:44.196278 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:44.196285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:44.196438 (+ 153us) server_negotiation.cc:407] Connection header received
0929 02:55:44.196591 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:44.196596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:44.196671 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:44.196749 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:44.197407 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35724: BlockingRecv error: recv got EOF from 127.0.0.1:35724 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 02:55:44.197741 15433 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790650496826590-15433-0/krb5kdc/krb5cc)
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:44.199137 15433 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:44.200357 15433 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:55:44.214301 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:44.201586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.15.18.65:35607)
0929 02:55:44.201792 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:44.201798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:44.201997 (+ 199us) server_negotiation.cc:407] Connection header received
0929 02:55:44.202114 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:44.202118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:44.202175 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:44.202256 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:44.203132 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.203930 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.205229 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.205626 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.205757 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:44.209270 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:44.209275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:44.209279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:44.209312 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:44.211873 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.212518 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.212522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.212524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.212674 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.213012 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.213015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.213017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.213325 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:44.213553 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:44.213787 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 02:55:44.213970 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:55:44.215888 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 21114
I20260929 02:55:44.227753 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:44.216618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.15.18.65:35607)
0929 02:55:44.216788 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:44.216795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:44.217107 (+ 312us) server_negotiation.cc:407] Connection header received
0929 02:55:44.217295 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:44.217300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:44.217370 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:44.217452 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:44.218405 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.219070 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.220299 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.220743 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.220850 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:44.222847 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:44.222852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:44.222855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:44.222886 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:44.225266 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.225941 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.225947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.225950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.226085 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.226436 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.226439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.226441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.226697 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:44.226941 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:44.227231 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 02:55:44.227439 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:55:44.228394 21237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:44.310487 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 21114
I20260929 02:55:44.323244 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 21248
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:55:44.337816 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:44.324153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.15.18.66:37035)
0929 02:55:44.324675 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:44.324683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:44.324708 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:55:44.324771 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:44.324778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:44.325002 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:44.325126 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:44.326027 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.327015 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.328291 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.328829 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.328931 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:44.332541 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:44.332552 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:44.332568 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:44.332600 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:44.335276 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.335996 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.336004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.336006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.336127 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.336445 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.336447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.336449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.336765 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:44.336987 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:44.337271 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 02:55:44.337491 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":293,"threads_started":1}
I20260929 02:55:44.339067 21371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:44.420955 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 21248
I20260929 02:55:44.433372 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 21381
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:55:44.448724 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:44.434418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.15.18.67:41333)
0929 02:55:44.435062 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:44.435069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:44.435095 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:55:44.435153 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:44.435159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:44.435362 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:44.435488 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:44.436507 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.437501 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.438727 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:44.439280 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:44.439379 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:44.443314 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:44.443325 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:44.443342 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:44.443384 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:44.446065 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.446711 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.446718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.446720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.446849 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:44.447196 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:44.447201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:44.447203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:44.447640 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:44.447929 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:44.448187 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 02:55:44.448431 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":326,"threads_started":1}
I20260929 02:55:44.450075 21504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:44.532219 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 21381
I20260929 02:55:44.544135 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 21020
I20260929 02:55:44.545275 21082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:44.646651 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 21020
2026-09-29T02:55:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3760 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21536](info): set up 2 sockets
Sep 29 02:55:44 dist-test-slave-q5f0 krb5kdc[21536](info): commencing operation
krb5kdc: starting...
W20260929 02:55:45.644433 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:55:45 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.126.keytab.
Entry for principal oryx/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.126.keytab.
I20260929 02:55:45.839707 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43925
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.126.keytab
--principal=oryx/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:46.150692 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:46.150997 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:46.151109 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:46.158474 21549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:46.158599 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:46.158648 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:46.158727 21549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:46.158779 21549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:46.168817 21549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.126.keytab
--principal=oryx/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:46.170197 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:46.171495 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:46.182895 21554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:46.182967 21557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:46.182937 21555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:46.183880 21549 server_base.cc:1034] running on GCE node
I20260929 02:55:46.184695 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:46.186161 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:46.187371 21549 hybrid_clock.cc:648] HybridClock initialized: now 1790650546187318 us; error 36 us; skew 500 ppm
I20260929 02:55:46.187796 21549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:46 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650546, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:46.194317 21549 init.cc:377] Logged in from keytab as oryx/127.15.18.126@KRBTEST.COM (short username oryx)
I20260929 02:55:46.195987 21549 webserver.cc:460] Webserver started at http://127.15.18.126:34413/ using document root <none> and password file <none>
I20260929 02:55:46.196807 21549 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:46.196939 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:46.197285 21549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:46.200557 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "f3079672211e419b8cf87e10fb5134a5"
format_stamp: "Formatted at 2026-09-29 02:55:46 on dist-test-slave-q5f0"
server_key: "554be22d85853ffc3c20564f6462165a"
server_key_iv: "0e56ba700a78606c987f19a18ecb6e38"
server_key_version: "encryptionkey@0"
I20260929 02:55:46.201366 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "f3079672211e419b8cf87e10fb5134a5"
format_stamp: "Formatted at 2026-09-29 02:55:46 on dist-test-slave-q5f0"
server_key: "554be22d85853ffc3c20564f6462165a"
server_key_iv: "0e56ba700a78606c987f19a18ecb6e38"
server_key_version: "encryptionkey@0"
I20260929 02:55:46.206956 21549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:46.210515 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:46.212100 21549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:46.212287 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "f3079672211e419b8cf87e10fb5134a5"
format_stamp: "Formatted at 2026-09-29 02:55:46 on dist-test-slave-q5f0"
server_key: "554be22d85853ffc3c20564f6462165a"
server_key_iv: "0e56ba700a78606c987f19a18ecb6e38"
server_key_version: "encryptionkey@0"
I20260929 02:55:46.212484 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:46.231118 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:46.235792 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:46.236097 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:46.253618 21549 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:43925
I20260929 02:55:46.253655 21615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:43925 every 8 connection(s)
I20260929 02:55:46.254954 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:46.258687 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 21549
I20260929 02:55:46.258886 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:46.258903 21616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:46.259485 15433 external_mini_cluster.cc:1428] Setting key 7f61c807afaf15d6160a7c654e483c70
I20260929 02:55:46.273379 21616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5: Bootstrap starting.
Sep 29 02:55:46 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:46.277197 21616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:46.278581 21616 log.cc:824] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:46.282307 21616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5: No bootstrap required, opened a new log
I20260929 02:55:46.284229 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:46.261461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.15.18.126:43925)
0929 02:55:46.262296 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:46.262314 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:46.262369 (+ 55us) server_negotiation.cc:407] Connection header received
0929 02:55:46.263986 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:46.264018 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:46.264561 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:46.265033 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:46.266347 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:46.268322 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:46.269766 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:46.270933 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:46.271142 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:46.275302 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:46.275313 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:46.275333 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:46.275393 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:46.279238 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:46.280175 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:46.280184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:46.280193 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:46.280412 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:46.280865 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:46.280870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:46.280873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:46.281595 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:46.281950 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:46.282335 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 02:55:46.282832 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":349,"threads_started":1}
I20260929 02:55:46.290781 21616 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } }
I20260929 02:55:46.291142 21616 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:46.291222 21616 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3079672211e419b8cf87e10fb5134a5, State: Initialized, Role: FOLLOWER
I20260929 02:55:46.291805 21616 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } }
I20260929 02:55:46.292001 21616 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:46.292096 21616 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:46.292222 21616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:46.294569 21616 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } }
I20260929 02:55:46.295046 21616 leader_election.cc:302] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3079672211e419b8cf87e10fb5134a5; no voters:
I20260929 02:55:46.295644 21616 leader_election.cc:288] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:46.295794 21621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:46.296418 21621 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 1 LEADER]: Becoming Leader. State: Replica: f3079672211e419b8cf87e10fb5134a5, State: Running, Role: LEADER
I20260929 02:55:46.296952 21621 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } }
I20260929 02:55:46.297595 21616 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:46.300014 21622 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } }
I20260929 02:55:46.300077 21623 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3079672211e419b8cf87e10fb5134a5. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } }
I20260929 02:55:46.300391 21622 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:46.300431 21623 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:46.304224 21627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:46.309592 21627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:46.319023 21627 catalog_manager.cc:1347] Generated new cluster ID: fb387822bc0a47e7ab62ee0e6ba510b7
I20260929 02:55:46.319136 21627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:46.328150 21627 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:46.329490 21627 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:46.345314 21627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5: Generated new TSK 0
I20260929 02:55:46.346169 21627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab.
Entry for principal oryx/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab.
I20260929 02:55:46.403192 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43925
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab
--principal=oryx/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:46.726960 21642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:46.727234 21642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:46.727353 21642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:46.734638 21642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:46.734745 21642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:46.734898 21642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:46.744777 21642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab
--principal=oryx/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:46.746083 21642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:46.747409 21642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:46.759980 21647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:46.759928 21648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:46.759939 21650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:46.761042 21642 server_base.cc:1034] running on GCE node
I20260929 02:55:46.761655 21642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:46.762609 21642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:46.763798 21642 hybrid_clock.cc:648] HybridClock initialized: now 1790650546763735 us; error 54 us; skew 500 ppm
I20260929 02:55:46.764112 21642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:46 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650546, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:46.767280 21642 init.cc:377] Logged in from keytab as oryx/127.15.18.65@KRBTEST.COM (short username oryx)
I20260929 02:55:46.768824 21642 webserver.cc:460] Webserver started at http://127.15.18.65:39913/ using document root <none> and password file <none>
I20260929 02:55:46.769518 21642 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:46.769634 21642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:46.769917 21642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:46.772783 21642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "0e0628f790c74c0bad86b970bbf2bd80"
format_stamp: "Formatted at 2026-09-29 02:55:46 on dist-test-slave-q5f0"
server_key: "f62d0b1fccebeca2d5c04e809d5a5ae5"
server_key_iv: "909038b7ddc13f7e1561cc7ac77944c2"
server_key_version: "encryptionkey@0"
I20260929 02:55:46.773535 21642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "0e0628f790c74c0bad86b970bbf2bd80"
format_stamp: "Formatted at 2026-09-29 02:55:46 on dist-test-slave-q5f0"
server_key: "f62d0b1fccebeca2d5c04e809d5a5ae5"
server_key_iv: "909038b7ddc13f7e1561cc7ac77944c2"
server_key_version: "encryptionkey@0"
I20260929 02:55:46.778710 21642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 02:55:46.782297 21657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:46.783828 21642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:46.783986 21642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "0e0628f790c74c0bad86b970bbf2bd80"
format_stamp: "Formatted at 2026-09-29 02:55:46 on dist-test-slave-q5f0"
server_key: "f62d0b1fccebeca2d5c04e809d5a5ae5"
server_key_iv: "909038b7ddc13f7e1561cc7ac77944c2"
server_key_version: "encryptionkey@0"
I20260929 02:55:46.784165 21642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:46.816320 21642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:46.820554 21642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:46.820817 21642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:46.821767 21642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:46.823081 21642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:46.823153 21642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:46.823276 21642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:46.823328 21642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:46.853730 21642 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:40765
I20260929 02:55:46.853771 21769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:40765 every 8 connection(s)
I20260929 02:55:46.855007 21642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:46.856289 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 21642
I20260929 02:55:46.856474 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:46.857169 15433 external_mini_cluster.cc:1428] Setting key dc072135e6c1c688ffea64aab77070cf
Sep 29 02:55:46 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650546, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:46.875962 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:46.857514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:37855 (local address 127.15.18.126:43925)
0929 02:55:46.858266 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:46.858272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:46.859218 (+ 946us) server_negotiation.cc:407] Connection header received
0929 02:55:46.861025 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:46.861030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:46.861113 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:46.861220 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:46.862631 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:46.863616 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:46.865814 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:46.866203 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:46.866318 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:46.870264 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:46.870269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:46.870273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:46.870306 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:46.872662 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:46.873365 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:46.873371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:46.873374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:46.873487 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:46.873882 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:46.873887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:46.873891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:46.874147 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:46.874273 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:46.875132 (+ 859us) server_negotiation.cc:299] Negotiation successful
0929 02:55:46.875472 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":522,"threads_started":1}
WARNING: no policy specified for oryx/127.15.18.66@KRBTEST.COM; defaulting to no policy
I20260929 02:55:46.876722 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:46.858272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.65:37855)
0929 02:55:46.859060 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:46.859117 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:46.860763 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:46.861410 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:46.861418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:46.861929 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:46.862477 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:46.862494 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:46.863839 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:46.863844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:46.865621 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:46.865637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:46.866453 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:46.866460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:46.866755 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:46.867484 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:46.867549 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:46.870024 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:46.872801 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:46.872811 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:46.872830 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:46.873242 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:46.873588 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:46.873593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:46.873597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:46.873778 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:46.874273 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:46.874282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:46.874621 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 02:55:46.874921 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 02:55:46.875537 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":311,"threads_started":1}
I20260929 02:55:46.878166 21770 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43925
I20260929 02:55:46.878461 21770 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.66@KRBTEST.COM" created.
I20260929 02:55:46.879287 21770 heartbeater.cc:511] Master 127.15.18.126:43925 requested a full tablet report, sending...
I20260929 02:55:46.881464 21581 ts_manager.cc:194] Registered new tserver with Master: 0e0628f790c74c0bad86b970bbf2bd80 (127.15.18.65:40765)
I20260929 02:55:46.882977 21581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:37855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab.
Entry for principal oryx/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab.
I20260929 02:55:46.903244 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43925
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab
--principal=oryx/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:47.220500 21776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:47.220785 21776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:47.220893 21776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:47.228219 21776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:47.228338 21776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:47.228451 21776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:47.238179 21776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab
--principal=oryx/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:47.239408 21776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:47.240759 21776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:47.253453 21781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:47.253764 21784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:47.255746 21782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:47.256549 21776 server_base.cc:1034] running on GCE node
I20260929 02:55:47.257155 21776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:47.258175 21776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:47.259398 21776 hybrid_clock.cc:648] HybridClock initialized: now 1790650547259356 us; error 46 us; skew 500 ppm
I20260929 02:55:47.259732 21776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:47 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650547, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:47.262893 21776 init.cc:377] Logged in from keytab as oryx/127.15.18.66@KRBTEST.COM (short username oryx)
I20260929 02:55:47.264484 21776 webserver.cc:460] Webserver started at http://127.15.18.66:41505/ using document root <none> and password file <none>
I20260929 02:55:47.265162 21776 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:47.265262 21776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:47.265537 21776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:47.268469 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "a4258656104f48bfbebd1b16b87233d0"
format_stamp: "Formatted at 2026-09-29 02:55:47 on dist-test-slave-q5f0"
server_key: "8ae1f19bcf2b08e4bd6bcc90e62064bd"
server_key_iv: "6f2a72233917d3d0f5f0ada5e0a24045"
server_key_version: "encryptionkey@0"
I20260929 02:55:47.269208 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "a4258656104f48bfbebd1b16b87233d0"
format_stamp: "Formatted at 2026-09-29 02:55:47 on dist-test-slave-q5f0"
server_key: "8ae1f19bcf2b08e4bd6bcc90e62064bd"
server_key_iv: "6f2a72233917d3d0f5f0ada5e0a24045"
server_key_version: "encryptionkey@0"
I20260929 02:55:47.274379 21776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:47.277850 21791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:47.279186 21776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 02:55:47.279328 21776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "a4258656104f48bfbebd1b16b87233d0"
format_stamp: "Formatted at 2026-09-29 02:55:47 on dist-test-slave-q5f0"
server_key: "8ae1f19bcf2b08e4bd6bcc90e62064bd"
server_key_iv: "6f2a72233917d3d0f5f0ada5e0a24045"
server_key_version: "encryptionkey@0"
I20260929 02:55:47.279548 21776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:47.309551 21776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:47.314040 21776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:47.314316 21776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:47.315294 21776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:47.316632 21776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:47.316716 21776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:47.316833 21776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:47.316893 21776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:47.346964 21776 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:45497
I20260929 02:55:47.346997 21903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:45497 every 8 connection(s)
I20260929 02:55:47.348377 21776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:47.352626 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 21776
I20260929 02:55:47.352866 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:47.353451 15433 external_mini_cluster.cc:1428] Setting key a0cbdbb1e50122ce9741e6bacc0a4e97
Sep 29 02:55:47 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650547, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:47.372172 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:47.351091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:32889 (local address 127.15.18.126:43925)
0929 02:55:47.351257 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:47.351262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:47.352635 (+ 1373us) server_negotiation.cc:407] Connection header received
0929 02:55:47.355011 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:47.355017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:47.355092 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:47.355195 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:47.357557 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:47.358247 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:47.360313 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:47.360728 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:47.360824 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:47.364625 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:47.364633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:47.364637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:47.364693 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:47.368077 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:47.368901 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:47.368907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:47.368910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:47.369082 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:47.370071 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:47.370077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:47.370080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:47.370471 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:47.370628 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:47.371463 (+ 835us) server_negotiation.cc:299] Negotiation successful
0929 02:55:47.371762 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:55:47.372944 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:47.351579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.66:32889)
0929 02:55:47.352420 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:47.352482 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:47.354738 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:47.355829 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:47.355841 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:47.356566 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:47.357358 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:47.357384 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:47.358399 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:47.358405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:47.360154 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:47.360168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:47.360934 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:47.360940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:47.361230 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:47.361924 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:47.361953 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:47.364302 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:47.368269 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:47.368278 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:47.368293 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:47.368757 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:47.369218 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:47.369222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:47.369224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:47.369448 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:47.370617 (+ 1169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:47.370624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:47.370999 (+ 375us) client_negotiation.cc:769] Sending connection context
0929 02:55:47.371303 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 02:55:47.371791 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":300,"threads_started":1}
I20260929 02:55:47.374485 21904 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43925
I20260929 02:55:47.374882 21904 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.15.18.67@KRBTEST.COM; defaulting to no policy
I20260929 02:55:47.375967 21904 heartbeater.cc:511] Master 127.15.18.126:43925 requested a full tablet report, sending...
I20260929 02:55:47.377805 21581 ts_manager.cc:194] Registered new tserver with Master: a4258656104f48bfbebd1b16b87233d0 (127.15.18.66:45497)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.67@KRBTEST.COM" created.
I20260929 02:55:47.378810 21581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:32889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab.
Entry for principal oryx/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab.
I20260929 02:55:47.402706 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43925
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab
--principal=oryx/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:47.728382 21909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:47.728657 21909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:47.728760 21909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:47.735894 21909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:47.736006 21909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:47.736129 21909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:47.745875 21909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab
--principal=oryx/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:47.747104 21909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:47.748409 21909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:47.761119 21914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:47.761089 21915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:47.761207 21917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:47.761703 21909 server_base.cc:1034] running on GCE node
I20260929 02:55:47.762343 21909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:47.763291 21909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:47.764513 21909 hybrid_clock.cc:648] HybridClock initialized: now 1790650547764455 us; error 46 us; skew 500 ppm
I20260929 02:55:47.764847 21909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:47 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650547, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:47.767848 21909 init.cc:377] Logged in from keytab as oryx/127.15.18.67@KRBTEST.COM (short username oryx)
I20260929 02:55:47.769168 21909 webserver.cc:460] Webserver started at http://127.15.18.67:38249/ using document root <none> and password file <none>
I20260929 02:55:47.769843 21909 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:47.769953 21909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:47.770249 21909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:47.773128 21909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "c45d1de6e9294aee87b262d36df0307f"
format_stamp: "Formatted at 2026-09-29 02:55:47 on dist-test-slave-q5f0"
server_key: "b71f6dc15114b59a67c209122a165764"
server_key_iv: "3b39679c6346744e757d9b60ef9d68fb"
server_key_version: "encryptionkey@0"
I20260929 02:55:47.773905 21909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "c45d1de6e9294aee87b262d36df0307f"
format_stamp: "Formatted at 2026-09-29 02:55:47 on dist-test-slave-q5f0"
server_key: "b71f6dc15114b59a67c209122a165764"
server_key_iv: "3b39679c6346744e757d9b60ef9d68fb"
server_key_version: "encryptionkey@0"
I20260929 02:55:47.779048 21909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:55:47.782539 21924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:47.784065 21909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:55:47.784227 21909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "c45d1de6e9294aee87b262d36df0307f"
format_stamp: "Formatted at 2026-09-29 02:55:47 on dist-test-slave-q5f0"
server_key: "b71f6dc15114b59a67c209122a165764"
server_key_iv: "3b39679c6346744e757d9b60ef9d68fb"
server_key_version: "encryptionkey@0"
I20260929 02:55:47.784403 21909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:47.804471 21909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:47.808887 21909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:47.809154 21909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:47.810154 21909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:47.811542 21909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:47.811617 21909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:47.811743 21909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:47.811795 21909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:47.841562 21909 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:45099
I20260929 02:55:47.841645 22036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:45099 every 8 connection(s)
I20260929 02:55:47.842798 21909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:47.852133 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 21909
I20260929 02:55:47.852286 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:47.852717 15433 external_mini_cluster.cc:1428] Setting key 9d3547eb7b3e9fb04de82338003c7d4e
Sep 29 02:55:47 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650547, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:47.862555 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:47.845150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:40807 (local address 127.15.18.126:43925)
0929 02:55:47.845363 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:47.845369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:47.846485 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 02:55:47.848296 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:47.848303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:47.848377 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:47.848474 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:47.850138 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:47.850913 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:47.852576 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:47.853048 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:47.853174 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:47.857016 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:47.857022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:47.857027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:47.857068 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:47.859979 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:47.860660 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:47.860663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:47.860666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:47.860785 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:47.861145 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:47.861148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:47.861150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:47.861415 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:47.861518 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:47.862146 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 02:55:47.862304 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:55:47.863588 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:47.845632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.67:40807)
0929 02:55:47.846331 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:47.846374 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:47.848010 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:47.848761 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:47.848770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:47.849324 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:47.849928 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:47.849947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:47.851072 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:47.851076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:47.852436 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:47.852450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:47.853303 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:47.853312 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:47.853626 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:47.854369 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:47.854407 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:47.856772 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:47.860113 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:47.860120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:47.860133 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:47.860561 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:47.860869 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:47.860873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:47.860877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:47.861051 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:47.861505 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:47.861512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:47.861821 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 02:55:47.862043 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 02:55:47.862437 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":269,"threads_started":1}
I20260929 02:55:47.865024 22037 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43925
I20260929 02:55:47.865347 22037 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:47.866070 22037 heartbeater.cc:511] Master 127.15.18.126:43925 requested a full tablet report, sending...
I20260929 02:55:47.867543 21581 ts_manager.cc:194] Registered new tserver with Master: c45d1de6e9294aee87b262d36df0307f (127.15.18.67:45099)
I20260929 02:55:47.868454 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:55:47.868456 21581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:40807
I20260929 02:55:47.885855 21770 heartbeater.cc:503] Master 127.15.18.126:43925 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.15.18.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.125.keytab.
Entry for principal oryx/127.15.18.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.125.keytab.
I20260929 02:55:47.915786 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.125:33733
--webserver_interface=127.15.18.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.125.keytab
--principal=oryx/127.15.18.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43925,127.15.18.125:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:48.234874 22042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:48.235188 22042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:48.235316 22042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:48.242868 22042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:48.242995 22042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:48.243072 22042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:48.243149 22042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:48.243213 22042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:48.253443 22042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43925,127.15.18.125:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.125.keytab
--principal=oryx/127.15.18.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.125:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.18.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
W20260929 02:55:48.253816 22042 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.18.126:43925,127.15.18.125:33733'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 02:55:48.254850 22042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:48.256316 22042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:48.268098 22047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:48.268144 22048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:48.268157 22050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:48.269125 22042 server_base.cc:1034] running on GCE node
I20260929 02:55:48.269948 22042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:48.271422 22042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:48.272676 22042 hybrid_clock.cc:648] HybridClock initialized: now 1790650548272612 us; error 36 us; skew 500 ppm
I20260929 02:55:48.273026 22042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:48 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650548, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:48.276072 22042 init.cc:377] Logged in from keytab as oryx/127.15.18.125@KRBTEST.COM (short username oryx)
I20260929 02:55:48.277585 22042 webserver.cc:460] Webserver started at http://127.15.18.125:44513/ using document root <none> and password file <none>
I20260929 02:55:48.278254 22042 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:48.278362 22042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:48.278656 22042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:48.281576 22042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data/instance:
uuid: "943a13ef3a5045c582fa14727c05da9f"
format_stamp: "Formatted at 2026-09-29 02:55:48 on dist-test-slave-q5f0"
server_key: "fd14909ed4b1e8523021501a7fb5c870"
server_key_iv: "3f814b2e92bad9ecdfcbf2e4dc1e4d1d"
server_key_version: "encryptionkey@0"
I20260929 02:55:48.282352 22042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal/instance:
uuid: "943a13ef3a5045c582fa14727c05da9f"
format_stamp: "Formatted at 2026-09-29 02:55:48 on dist-test-slave-q5f0"
server_key: "fd14909ed4b1e8523021501a7fb5c870"
server_key_iv: "3f814b2e92bad9ecdfcbf2e4dc1e4d1d"
server_key_version: "encryptionkey@0"
I20260929 02:55:48.287614 22042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:55:48.291182 22057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:48.292644 22042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:48.292797 22042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal
uuid: "943a13ef3a5045c582fa14727c05da9f"
format_stamp: "Formatted at 2026-09-29 02:55:48 on dist-test-slave-q5f0"
server_key: "fd14909ed4b1e8523021501a7fb5c870"
server_key_iv: "3f814b2e92bad9ecdfcbf2e4dc1e4d1d"
server_key_version: "encryptionkey@0"
I20260929 02:55:48.293000 22042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:48.330529 22042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:48.334645 22042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:48.334890 22042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:48.352294 22042 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.125:33733
I20260929 02:55:48.352260 22108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.125:33733 every 8 connection(s)
I20260929 02:55:48.353832 22042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
I20260929 02:55:48.355139 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 22042
I20260929 02:55:48.358366 22109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:48.365160 22109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:55:48 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.125@KRBTEST.COM
Sep 29 02:55:48 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650548, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.125@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:48.377635 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.357360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.15.18.125:33733)
0929 02:55:48.358102 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:48.358114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:48.358154 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:55:48.359755 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:48.359783 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:48.360155 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:48.360506 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:48.361952 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.363278 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.364682 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.365348 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.365528 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.369930 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:48.369936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:48.369946 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:48.369993 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:48.373097 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.373800 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.373810 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.373819 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.374015 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.374397 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.374401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.374404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.375027 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:48.375280 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:48.375886 (+ 606us) server_negotiation.cc:299] Negotiation successful
0929 02:55:48.376372 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":346,"threads_started":1}
I20260929 02:55:48.381871 21904 heartbeater.cc:503] Master 127.15.18.126:43925 was elected leader, sending a full tablet report...
I20260929 02:55:48.383242 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.367470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:59532)
0929 02:55:48.368144 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:48.368184 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:48.368317 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:48.368990 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:48.368997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:48.369811 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:48.370121 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.370130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.371569 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.371575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.372731 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.372743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.373754 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.373761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.373926 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.374489 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:48.374506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:48.376924 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:48.380531 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.380541 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.380548 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.381037 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.381413 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.381417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.381421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.381607 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.382272 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:48.382279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:48.382455 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 02:55:48.382640 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 02:55:48.382923 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":275,"threads_started":1}
I20260929 02:55:48.383329 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.367584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.15.18.126:43925)
0929 02:55:48.368663 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:48.368667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:48.368682 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:55:48.368736 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:48.368740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:48.368801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:48.368897 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:48.370274 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.371372 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.372885 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.373564 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.373685 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.377113 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:48.377119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:48.377124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:48.377164 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:48.380386 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.381146 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.381150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.381152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.381293 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.381727 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.381732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.381735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.382140 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:48.382275 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:48.382772 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 02:55:48.382967 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":902,"threads_started":1}
I20260929 02:55:48.384873 22109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:55:48 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650548, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.125@KRBTEST.COM for oryx/127.15.18.125@KRBTEST.COM
I20260929 02:55:48.396749 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.385633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:33733 (local address 127.0.0.1:43350)
0929 02:55:48.385834 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:48.385850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:48.385941 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:48.386166 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:48.386170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:48.386463 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:48.386587 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.386594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.387579 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.387583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.388525 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.388539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.389273 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.389278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.389413 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.389900 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:48.389916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:48.391884 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:48.394608 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.394612 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.394615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.394977 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.395334 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.395338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.395340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.395467 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.396059 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:48.396062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:48.396186 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 02:55:48.396301 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 02:55:48.396511 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 02:55:48.397013 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.385675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.15.18.125:33733)
0929 02:55:48.385836 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:48.385842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:48.385901 (+ 59us) server_negotiation.cc:407] Connection header received
0929 02:55:48.386011 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:48.386014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:48.386064 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:48.386131 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:48.386695 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.387390 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.388683 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.389119 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.389226 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.392062 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:48.392067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:48.392071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:48.392099 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:48.394460 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.395082 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.395086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.395089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.395225 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.395576 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.395581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.395584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.395862 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:48.396078 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:48.396394 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 02:55:48.396645 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":56}
I20260929 02:55:48.401623 22109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Bootstrap starting.
I20260929 02:55:48.404791 22109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:48.405843 22109 log.cc:824] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:48.408610 22109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: No bootstrap required, opened a new log
I20260929 02:55:48.415364 22109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } }
I20260929 02:55:48.415738 22109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:48.415827 22109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 943a13ef3a5045c582fa14727c05da9f, State: Initialized, Role: FOLLOWER
I20260929 02:55:48.416391 22109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } }
I20260929 02:55:48.417245 22116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } } }
I20260929 02:55:48.417549 22116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:55:48.418182 22109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:55:48.425999 22127 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:55:48.426126 22127 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:55:48.428520 22042 master_runner.cc:428] Detected that this master 943a13ef3a5045c582fa14727c05da9f is joining an existing cluster
I20260929 02:55:48.428607 22042 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.18.125:33733
I20260929 02:55:48.449798 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.433855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:59546)
0929 02:55:48.435947 (+ 2092us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:48.435965 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:48.436165 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:48.436670 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:48.436674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:48.437304 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:48.437688 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.437700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.438977 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.438981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.440124 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.440136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.441021 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.441027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.441205 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.441834 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:48.441851 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:48.442689 (+ 838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:48.446605 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.446610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.446614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.447119 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.447615 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.447620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.447624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.447814 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.448479 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:48.448484 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:48.448711 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 02:55:48.448869 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 02:55:48.449212 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1964,"thread_start_us":1814,"threads_started":1}
I20260929 02:55:48.451406 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.433879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.15.18.126:43925)
0929 02:55:48.434048 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:48.434054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:48.436055 (+ 2001us) server_negotiation.cc:407] Connection header received
0929 02:55:48.436313 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:48.436319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:48.436425 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:48.436485 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:48.437846 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.438811 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.440255 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.440847 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.440962 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.442886 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:48.442893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:48.442898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:48.442940 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:48.446452 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.447286 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.447292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.447296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.447451 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.447968 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.447973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.447976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.448346 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:48.450624 (+ 2278us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:48.450776 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 02:55:48.451040 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:55:48.454797 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.434130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.15.18.125:33733)
0929 02:55:48.434345 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:48.434351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:48.440798 (+ 6447us) server_negotiation.cc:407] Connection header received
0929 02:55:48.440946 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:48.440951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:48.441032 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:48.441138 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:48.442638 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.443991 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.445509 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.446156 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.446257 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.448395 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:48.448401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:48.448405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:48.448434 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:48.451943 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.452741 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.452746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.452749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.452891 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.453250 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.453254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.453257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.453619 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:48.453764 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:48.454226 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 02:55:48.454429 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":61}
I20260929 02:55:48.456591 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.436156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:33733 (local address 127.0.0.1:43354)
0929 02:55:48.440691 (+ 4535us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:48.440709 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:48.440827 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:48.441319 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:48.441324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:48.442102 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:48.442471 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.442484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.444142 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.444145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.445371 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.445385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.446533 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.446538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.446687 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.447249 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:48.447269 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:48.448236 (+ 967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:48.452097 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.452101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.452105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.452618 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.453006 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.453010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.453013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.453149 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.453844 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:48.453849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:48.453987 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 02:55:48.455778 (+ 1791us) client_negotiation.cc:240] Negotiation successful
0929 02:55:48.456105 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4411,"thread_start_us":4217,"threads_started":1}
WARNING: no policy specified for oryx/127.15.18.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.124@KRBTEST.COM" created.
Sep 29 02:55:48 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650546, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.126@KRBTEST.COM for oryx/127.15.18.125@KRBTEST.COM
I20260929 02:55:48.477510 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.459277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.15.18.125:33733)
0929 02:55:48.459433 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:48.459438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:48.459753 (+ 315us) server_negotiation.cc:407] Connection header received
0929 02:55:48.460133 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:48.460139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:48.460222 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:48.460302 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:48.461730 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.463000 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.464921 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.465597 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.465750 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.470277 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:48.470285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:48.470289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:48.470333 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:48.474117 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.475100 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.475106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.475110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.475285 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.475785 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.475790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.475793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.476147 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:48.476304 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:48.476874 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 02:55:48.477106 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:55:48.477594 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.458992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:33733 (local address 127.0.0.1:43366)
0929 02:55:48.459635 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:48.459679 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:48.459880 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:48.460369 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:48.460375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:48.461190 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:48.461488 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.461500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.463134 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.463139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.464741 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.464757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.465792 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.465799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.466001 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.466744 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:48.466768 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:48.470029 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:48.474339 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.474355 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.474363 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.474932 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.475415 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.475421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.475424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.475641 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.476285 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:48.476294 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:48.476526 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 02:55:48.476758 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 02:55:48.477128 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":440,"threads_started":1}
I20260929 02:55:48.479965 21581 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.18.125:33733: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "943a13ef3a5045c582fa14727c05da9f"
member_type: NON_VOTER
last_known_addr {
host: "127.15.18.125"
port: 33733
}
attrs {
promote: true
}
}
dest_uuid: "f3079672211e419b8cf87e10fb5134a5"
cas_config_opid_index: -1
I20260929 02:55:48.481290 21581 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } }
I20260929 02:55:48.487185 22084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:48.486423 22140 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 943a13ef3a5045c582fa14727c05da9f (127.15.18.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } } }
I20260929 02:55:48.491621 22084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Refusing update from remote peer f3079672211e419b8cf87e10fb5134a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 02:55:48.492071 22140 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.18.125:33733
I20260929 02:55:48.492817 22140 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:48.492794 22139 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } } }
I20260929 02:55:48.493099 22042 master.cc:537] Master@127.15.18.125:33733 shutting down...
I20260929 02:55:48.493266 22139 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.124.keytab.
Entry for principal oryx/127.15.18.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.124.keytab.
I20260929 02:55:48.495441 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.124:44447
--webserver_interface=127.15.18.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.124.keytab
--principal=oryx/127.15.18.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43925,127.15.18.125:33733,127.15.18.124:44447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:48.495765 21568 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 02:55:48.502215 21568 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 -> Peer 943a13ef3a5045c582fa14727c05da9f (127.15.18.125:33733): Couldn't send request to peer 943a13ef3a5045c582fa14727c05da9f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:55:48.509385 22042 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:48.509868 22042 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:48.510066 22042 tablet_replica.cc:330] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: stopping tablet replica
I20260929 02:55:48.516736 22042 master.cc:552] Master@127.15.18.125:33733 shutdown complete.
I20260929 02:55:48.516812 22042 master_runner.cc:305] Clearing existing system tablet
I20260929 02:55:48.517236 22042 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:48.517330 22042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:48.519820 22042 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:48.521317 22145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:48.521698 22042 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:48.521956 22042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 02:55:48.522050 22042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal
uuid: "943a13ef3a5045c582fa14727c05da9f"
format_stamp: "Formatted at 2026-09-29 02:55:48 on dist-test-slave-q5f0"
server_key: "fd14909ed4b1e8523021501a7fb5c870"
server_key_iv: "3f814b2e92bad9ecdfcbf2e4dc1e4d1d"
server_key_version: "encryptionkey@0"
I20260929 02:55:48.525453 22042 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:48.530386 22042 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:55:48.530561 22042 log.cc:1196] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 02:55:48.530941 22042 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Deleting consensus metadata
I20260929 02:55:48.531175 22042 master_runner.cc:315] Copying system tablet from 127.15.18.126:43925
I20260929 02:55:48.533411 22042 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: tablet copy: Beginning tablet copy session from remote peer at address 127.15.18.126:43925
I20260929 02:55:48.545356 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.534159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.15.18.126:43925)
0929 02:55:48.534369 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:48.534376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:48.534832 (+ 456us) server_negotiation.cc:407] Connection header received
0929 02:55:48.535035 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:48.535041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:48.535109 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:48.535191 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:48.536114 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.536883 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.538147 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.538558 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.538675 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.540393 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:48.540398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:48.540401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:48.540436 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:48.542844 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.543435 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.543441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.543444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.543623 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:48.544013 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:48.544018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:48.544021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:48.544287 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:48.544417 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:48.544781 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 02:55:48.544978 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:55:48.545506 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.534100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:59562)
0929 02:55:48.534726 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:48.534742 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:48.534885 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:48.535244 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:48.535249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:48.535711 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:48.535917 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.535930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.537047 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.537053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.538005 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.538016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.538699 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.538705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.538881 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.539443 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:48.539461 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:48.540203 (+ 742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:48.542966 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.542972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.542976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.543331 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.543779 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:48.543784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:48.543788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:48.543918 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:48.544400 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:48.544405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:48.544559 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 02:55:48.544680 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 02:55:48.545041 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":381,"threads_started":1}
I20260929 02:55:48.546365 21601 tablet_copy_service.cc:140] P f3079672211e419b8cf87e10fb5134a5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 943a13ef3a5045c582fa14727c05da9f ({username='oryx', principal='oryx/127.15.18.125@KRBTEST.COM'} at 127.0.0.1:59562)
I20260929 02:55:48.546595 21601 tablet_copy_service.cc:161] P f3079672211e419b8cf87e10fb5134a5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 943a13ef3a5045c582fa14727c05da9f at {username='oryx', principal='oryx/127.15.18.125@KRBTEST.COM'} at 127.0.0.1:59562: session id = 943a13ef3a5045c582fa14727c05da9f-00000000000000000000000000000000
I20260929 02:55:48.549641 21601 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:55:48.551424 22042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:55:48.555992 22042 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: tablet copy: Starting download of 0 data blocks...
I20260929 02:55:48.556317 22042 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: tablet copy: Starting download of 1 WAL segments...
I20260929 02:55:48.558732 22042 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:55:48.561903 21601 tablet_copy_service.cc:342] P f3079672211e419b8cf87e10fb5134a5: Request end of tablet copy session 943a13ef3a5045c582fa14727c05da9f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.18.125@KRBTEST.COM'} at 127.0.0.1:59562
I20260929 02:55:48.562078 21601 tablet_copy_service.cc:433] P f3079672211e419b8cf87e10fb5134a5: ending tablet copy session 943a13ef3a5045c582fa14727c05da9f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 943a13ef3a5045c582fa14727c05da9f
I20260929 02:55:48.571899 22042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:48.574357 22152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:48.580355 22153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:48.584852 22155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:48.587402 22042 server_base.cc:1034] running on GCE node
I20260929 02:55:48.587873 22042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:48.588649 22042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:48.589926 22042 hybrid_clock.cc:648] HybridClock initialized: now 1790650548589872 us; error 38 us; skew 500 ppm
I20260929 02:55:48.590137 22042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:48 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650548, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:48.591886 22042 init.cc:377] Logged in from keytab as oryx/127.15.18.125@KRBTEST.COM (short username oryx)
I20260929 02:55:48.593031 22042 webserver.cc:460] Webserver started at http://127.15.18.125:41867/ using document root <none> and password file <none>
I20260929 02:55:48.593328 22042 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:48.593397 22042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:48.595752 22042 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 02:55:48.597302 22162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:48.597711 22042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 02:55:48.597841 22042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal
uuid: "943a13ef3a5045c582fa14727c05da9f"
format_stamp: "Formatted at 2026-09-29 02:55:48 on dist-test-slave-q5f0"
server_key: "fd14909ed4b1e8523021501a7fb5c870"
server_key_iv: "3f814b2e92bad9ecdfcbf2e4dc1e4d1d"
server_key_version: "encryptionkey@0"
I20260929 02:55:48.597981 22042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:48.626996 22042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:48.630437 22042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:48.649636 22042 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.125:33733
I20260929 02:55:48.649655 22213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.125:33733 every 8 connection(s)
I20260929 02:55:48.650657 22042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
I20260929 02:55:48.656255 22214 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:55:48.660509 22214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Bootstrap starting.
I20260929 02:55:48.675441 22214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:55:48.675923 22214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Bootstrap complete.
I20260929 02:55:48.676764 22214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } }
I20260929 02:55:48.676981 22214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 943a13ef3a5045c582fa14727c05da9f, State: Initialized, Role: LEARNER
I20260929 02:55:48.677207 22214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } }
I20260929 02:55:48.677958 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } } }
I20260929 02:55:48.678323 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: This master's current role is: LEARNER
I20260929 02:55:48.678334 22214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:48.690935 22228 catalog_manager.cc:1259] Loaded cluster ID: fb387822bc0a47e7ab62ee0e6ba510b7
I20260929 02:55:48.691053 22228 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: loading cluster ID for follower catalog manager: success
I20260929 02:55:48.694955 22228 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: acquiring CA information for follower catalog manager: success
I20260929 02:55:48.697556 22228 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:55:48.871541 22037 heartbeater.cc:503] Master 127.15.18.126:43925 was elected leader, sending a full tablet report...
W20260929 02:55:48.882838 22143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:48.883142 22143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:48.883248 22143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:48.890825 22143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:48.890936 22143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:48.890981 22143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:48.891036 22143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:48.891076 22143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:48.901120 22143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43925,127.15.18.125:33733,127.15.18.124:44447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.124.keytab
--principal=oryx/127.15.18.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.124:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.18.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:48.902415 22143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:48.903802 22143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:48.915256 22233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:48.915304 22234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:48.915246 22236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:48.916050 22143 server_base.cc:1034] running on GCE node
I20260929 02:55:48.916851 22143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:48.918298 22143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:48.919492 22143 hybrid_clock.cc:648] HybridClock initialized: now 1790650548919439 us; error 53 us; skew 500 ppm
I20260929 02:55:48.919845 22143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:48 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650548, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:48.922745 22143 init.cc:377] Logged in from keytab as oryx/127.15.18.124@KRBTEST.COM (short username oryx)
I20260929 02:55:48.924186 22143 webserver.cc:460] Webserver started at http://127.15.18.124:39117/ using document root <none> and password file <none>
I20260929 02:55:48.924836 22143 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:48.924947 22143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:48.925228 22143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:48.928172 22143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data/instance:
uuid: "5fff2b419dcf43688659f1bdb5541007"
format_stamp: "Formatted at 2026-09-29 02:55:48 on dist-test-slave-q5f0"
server_key: "f4689debb803435bf695145c55a8b320"
server_key_iv: "6b78b2d8ba9f801e95138890a39608d3"
server_key_version: "encryptionkey@0"
I20260929 02:55:48.928910 22143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal/instance:
uuid: "5fff2b419dcf43688659f1bdb5541007"
format_stamp: "Formatted at 2026-09-29 02:55:48 on dist-test-slave-q5f0"
server_key: "f4689debb803435bf695145c55a8b320"
server_key_iv: "6b78b2d8ba9f801e95138890a39608d3"
server_key_version: "encryptionkey@0"
I20260929 02:55:48.934054 22143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:55:48.937502 22243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:48.938817 22143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:55:48.938977 22143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal
uuid: "5fff2b419dcf43688659f1bdb5541007"
format_stamp: "Formatted at 2026-09-29 02:55:48 on dist-test-slave-q5f0"
server_key: "f4689debb803435bf695145c55a8b320"
server_key_iv: "6b78b2d8ba9f801e95138890a39608d3"
server_key_version: "encryptionkey@0"
I20260929 02:55:48.939148 22143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:48.952818 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.947080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:33733 (local address 127.0.0.1:43372)
0929 02:55:48.947263 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:48.947279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:48.947378 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:48.948132 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:48.948137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:48.948165 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:48.948340 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.948349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.949632 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.949637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.950873 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:48.950885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.952055 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.952060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:48.952283 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.952334 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 02:55:48.952415 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:55:48.952551 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 02:55:48.952886 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.947277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.15.18.125:33733)
0929 02:55:48.947823 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:48.947828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:48.947843 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:55:48.947902 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:48.947905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:48.947970 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:48.948070 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:48.948487 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.949495 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.951019 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.951899 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.952011 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:48.952256 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:48.952455 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 02:55:48.952578 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":316,"threads_started":1}
I20260929 02:55:48.953879 22189 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 02:55:48.954421 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3079672211e419b8cf87e10fb5134a5. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } } }
I20260929 02:55:48.954662 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: This master's current role is: LEARNER
I20260929 02:55:48.959008 22143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:48.963138 22143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:48.963384 22143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:48.981025 22143 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.124:44447
I20260929 02:55:48.981047 22295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.124:44447 every 8 connection(s)
I20260929 02:55:48.982407 22143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
I20260929 02:55:48.984406 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 22143
I20260929 02:55:48.987246 22296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:48.995877 22296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:55:48 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.124@KRBTEST.COM
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650548, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.124@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:49.008791 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.986637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.15.18.124:44447)
0929 02:55:48.987583 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:48.987600 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:48.987939 (+ 339us) server_negotiation.cc:407] Connection header received
0929 02:55:48.989745 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:48.989767 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:48.990180 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:48.990531 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:48.992272 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.993604 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.995345 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:48.996097 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:48.996229 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.001053 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.001061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.001072 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.001109 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.004204 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.005031 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.005040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.005049 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.005259 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.005655 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.005660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.005664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.006392 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.006618 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.007207 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.007671 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":361,"threads_started":1}
I20260929 02:55:49.013949 22300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.998562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:59574)
0929 02:55:48.999383 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:48.999432 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:48.999629 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.000109 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.000117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.000754 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.001051 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.001068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.002267 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.002271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.003576 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.003589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.004325 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.004332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.004472 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.005060 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.005080 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.007934 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.011464 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.011471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.011476 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.012044 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.012371 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.012374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.012376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.012496 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.013109 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.013115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.013264 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.013422 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.013666 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":407,"threads_started":1}
I20260929 02:55:49.013996 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:48.998672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.15.18.126:43925)
0929 02:55:48.998856 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:48.998861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:48.999548 (+ 687us) server_negotiation.cc:407] Connection header received
0929 02:55:48.999779 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:48.999786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:48.999870 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:48.999973 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.001239 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.002119 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.003729 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.004108 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.004244 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.008134 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.008139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.008159 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.008198 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.011337 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.012157 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.012162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.012164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.012287 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.012682 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.012686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.012689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.013000 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.013128 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.013570 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.013770 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 02:55:49.015265 22296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650548, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.124@KRBTEST.COM for oryx/127.15.18.125@KRBTEST.COM
I20260929 02:55:49.027354 22300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.016047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:33733 (local address 127.0.0.1:43386)
0929 02:55:49.016240 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.016258 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.016360 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.016639 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.016643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.017071 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.017208 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.017216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.018166 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.018171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.019066 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.019077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.019759 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.019765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.019922 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.020422 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.020437 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.022364 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.025154 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.025158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.025160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.025528 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.025962 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.025966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.025969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.026110 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.026676 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.026679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.026801 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.026914 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.027122 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:55:49.027462 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.016059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.15.18.125:33733)
0929 02:55:49.016233 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.016239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.016304 (+ 65us) server_negotiation.cc:407] Connection header received
0929 02:55:49.016466 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.016472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.016533 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.016611 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.017337 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.018040 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.019193 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.019628 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.019742 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.022563 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.022568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.022575 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.022611 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.025032 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.025652 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.025657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.025659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.025830 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.026260 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.026265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.026267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.026547 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.026669 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.027027 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.027213 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:55:49.028456 22296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650548, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.124@KRBTEST.COM for oryx/127.15.18.124@KRBTEST.COM
I20260929 02:55:49.039937 22300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.029074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:44447 (local address 127.0.0.1:51054)
0929 02:55:49.029229 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.029246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.029347 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.029596 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.029600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.029936 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.030063 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.030071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.030951 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.030956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.031803 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.031813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.032534 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.032540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.032682 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.033164 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.033179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.035001 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.037792 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.037797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.037802 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.038114 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.038420 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.038424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.038428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.038560 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.039010 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.039014 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.039143 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.039252 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.039500 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:55:49.040001 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.029208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.15.18.124:44447)
0929 02:55:49.029335 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.029340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.029356 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:55:49.029445 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.029450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.029502 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.029571 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.030179 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.030843 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.031918 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.032387 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.032488 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.035147 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.035153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.035157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.035202 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.037683 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.038210 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.038215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.038219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.038328 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.038649 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.038654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.038658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.038911 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.039027 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.039335 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.039686 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":43}
I20260929 02:55:49.044775 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: Bootstrap starting.
I20260929 02:55:49.047760 22296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:49.048702 22296 log.cc:824] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:49.051178 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: No bootstrap required, opened a new log
I20260929 02:55:49.057911 22296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } }
I20260929 02:55:49.058238 22296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:49.058347 22296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fff2b419dcf43688659f1bdb5541007, State: Initialized, Role: FOLLOWER
I20260929 02:55:49.058919 22296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } }
I20260929 02:55:49.059778 22302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } } }
I20260929 02:55:49.060063 22302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:55:49.060710 22296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:55:49.069376 22313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:55:49.069535 22313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:55:49.073109 22143 master_runner.cc:428] Detected that this master 5fff2b419dcf43688659f1bdb5541007 is joining an existing cluster
I20260929 02:55:49.073232 22143 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.18.124:44447
I20260929 02:55:49.101459 22302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:49.101778 22302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } }
I20260929 02:55:49.102341 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.077035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.15.18.125:33733)
0929 02:55:49.077214 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.077220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.085276 (+ 8056us) server_negotiation.cc:407] Connection header received
0929 02:55:49.085491 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.085496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.085574 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.085639 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.087691 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.088786 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.091162 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.091857 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.091956 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.095062 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.095068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.095072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.095111 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.098806 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.099823 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.099829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.099832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.099996 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.100613 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.100617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.100620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.100959 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.101091 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.101804 (+ 713us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.102015 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:55:49.102371 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.077556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.15.18.124:44447)
0929 02:55:49.077911 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.077935 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.085107 (+ 7172us) server_negotiation.cc:407] Connection header received
0929 02:55:49.085441 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.085446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.085513 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.085582 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.087645 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.088807 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.091139 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.091868 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.091956 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.095048 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.095054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.095058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.095088 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.098803 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.099811 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.099816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.099818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.100116 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.100585 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.100590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.100596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.100965 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.101099 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.101768 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.101983 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260929 02:55:49.103392 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.077566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:33733 (local address 127.0.0.1:43390)
0929 02:55:49.085182 (+ 7616us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.085197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.085365 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.085824 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.085829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.086876 (+ 1047us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.087489 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.087498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.088990 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.088994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.090995 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.091008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.092365 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.092372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.092572 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.093236 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.093260 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.094885 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.099013 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.099017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.099020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.099680 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.100239 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.100242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.100245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.100501 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.101259 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.101263 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.101501 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.102504 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.102915 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7505,"mutex_wait_us":289,"spinlock_wait_cycles":72960,"thread_start_us":7330,"threads_started":1}
I20260929 02:55:49.103494 22302 leader_election.cc:288] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3079672211e419b8cf87e10fb5134a5 (127.15.18.126:43925), 943a13ef3a5045c582fa14727c05da9f (127.15.18.125:33733)
I20260929 02:55:49.103709 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.077148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:44447 (local address 127.0.0.1:51068)
0929 02:55:49.084952 (+ 7804us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.084981 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.085244 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.085782 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.085787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.086876 (+ 1089us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.087380 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.087450 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.089107 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.089113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.090993 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.091008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.092316 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.092323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.092516 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.093506 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.093543 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.094851 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.098978 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.098982 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.098986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.099716 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.100213 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.100218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.100221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.100473 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.101371 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.101375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.101506 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.102559 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.103082 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7628,"mutex_wait_us":377,"spinlock_wait_cycles":1792,"thread_start_us":7418,"threads_started":1}
I20260929 02:55:49.105141 21591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fff2b419dcf43688659f1bdb5541007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3079672211e419b8cf87e10fb5134a5" is_pre_election: true
I20260929 02:55:49.105260 22189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fff2b419dcf43688659f1bdb5541007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "943a13ef3a5045c582fa14727c05da9f" is_pre_election: true
I20260929 02:55:49.106233 22247 leader_election.cc:302] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fff2b419dcf43688659f1bdb5541007; no voters: 943a13ef3a5045c582fa14727c05da9f, f3079672211e419b8cf87e10fb5134a5
I20260929 02:55:49.106662 22302 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:55:49.108121 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.077742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.15.18.126:43925)
0929 02:55:49.077891 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.077896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.085311 (+ 7415us) server_negotiation.cc:407] Connection header received
0929 02:55:49.085691 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.085698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.085813 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.085942 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.087862 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.088982 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.091231 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.091933 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.092037 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.095074 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.095081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.095086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.095130 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.098989 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.099966 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.099972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.099976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.100136 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.100711 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.100717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.100720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.101059 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.101197 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.107549 (+ 6352us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.107792 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 02:55:49.108135 22319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.077104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:59584)
0929 02:55:49.085191 (+ 8087us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.085211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.085433 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.086284 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.086290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.087151 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.087669 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.087681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.089187 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.089192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.091056 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.091070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.092355 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.092362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.092570 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.093413 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.093435 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.094857 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.099196 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.099201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.099205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.099817 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.100282 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.100286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.100290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.100577 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.106926 (+ 6349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.106936 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.107169 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.107339 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.107673 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7261,"mutex_wait_us":151,"thread_start_us":6910,"threads_started":1}
I20260929 02:55:49.114915 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 21642
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650546, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.126@KRBTEST.COM for oryx/127.15.18.124@KRBTEST.COM
I20260929 02:55:49.127018 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.112668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:44447 (local address 127.0.0.1:51076)
0929 02:55:49.112828 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.112843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.112919 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.113359 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.113365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.113805 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.113990 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.114000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.115341 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.115347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.116832 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.116847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.117755 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.117760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.117906 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.118608 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.118630 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.120860 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.124462 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.124468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.124472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.124816 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.125235 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.125240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.125244 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.125425 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.126022 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.126027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.126197 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.126374 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.126649 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:55:49.127095 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.112734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.15.18.124:44447)
0929 02:55:49.112895 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.112900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.112918 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:49.113053 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.113059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.113138 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.113244 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.114126 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.115197 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.117017 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.117612 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.117729 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.121029 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.121034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.121037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.121063 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.124335 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.124920 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.124926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.124929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.125099 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.125542 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.125547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.125551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.125897 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.126046 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.126499 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.126726 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.65@KRBTEST.COM
I20260929 02:55:49.128877 21581 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.18.124:44447: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5fff2b419dcf43688659f1bdb5541007"
member_type: NON_VOTER
last_known_addr {
host: "127.15.18.124"
port: 44447
}
attrs {
promote: true
}
}
dest_uuid: "f3079672211e419b8cf87e10fb5134a5"
cas_config_opid_index: 5
I20260929 02:55:49.129887 21581 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } }
I20260929 02:55:49.132014 22139 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5fff2b419dcf43688659f1bdb5541007 (127.15.18.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.136425 22189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 LEARNER]: Refusing update from remote peer f3079672211e419b8cf87e10fb5134a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:55:49.137634 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.115938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.15.18.65:40765)
0929 02:55:49.117430 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.117439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.117468 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:55:49.117531 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.117540 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.117819 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.117990 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.119504 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.121050 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.122670 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.123571 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.123697 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.129164 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.129183 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.129202 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.129243 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.133051 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.133990 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.134001 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.134005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.134163 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.134633 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.134638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.134642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.135160 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.135461 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.136889 (+ 1428us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.137178 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":1154,"threads_started":1}
I20260929 02:55:49.137904 22271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:49.138995 21765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:49.140313 22324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 02:55:49.142203 22271 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 FOLLOWER]: Refusing update from remote peer f3079672211e419b8cf87e10fb5134a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:55:49.144819 22139 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.18.124:44447
I20260929 02:55:49.145484 22143 master.cc:537] Master@127.15.18.124:44447 shutting down...
I20260929 02:55:49.162894 22189 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5fff2b419dcf43688659f1bdb5541007 (127.15.18.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.166167 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.166469 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: This master's current role is: LEARNER
I20260929 02:55:49.174165 22143 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:49.174638 22143 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:49.173923 22323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 02:55:49.174834 22143 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: stopping tablet replica
I20260929 02:55:49.185343 22143 master.cc:552] Master@127.15.18.124:44447 shutdown complete.
I20260929 02:55:49.185514 22143 master_runner.cc:305] Clearing existing system tablet
I20260929 02:55:49.185951 22143 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:49.186131 22143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260929 02:55:49.193118 21565 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 -> Peer 5fff2b419dcf43688659f1bdb5541007 (127.15.18.124:44447): Couldn't send request to peer 5fff2b419dcf43688659f1bdb5541007. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:55:49.194360 22324 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.194633 22143 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.003s
I20260929 02:55:49.194708 22324 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:49.200750 22328 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5: attempting to promote NON_VOTER 943a13ef3a5045c582fa14727c05da9f to VOTER
I20260929 02:55:49.201766 22328 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } }
I20260929 02:55:49.204149 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.203616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:44447 (local address 127.0.0.1:51090)
0929 02:55:49.203851 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.203978 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 02:55:49.204195 22189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 LEARNER]: Refusing update from remote peer f3079672211e419b8cf87e10fb5134a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 02:55:49.204264 22139 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: NON_VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: true } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.204599 22139 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:49.205675 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.205207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:44447 (local address 127.0.0.1:51100)
0929 02:55:49.205394 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.205517 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:44447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 02:55:49.205933 21565 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 -> Peer 5fff2b419dcf43688659f1bdb5541007 (127.15.18.124:44447): Couldn't send request to peer 5fff2b419dcf43688659f1bdb5541007. Status: Network error: Client connection negotiation failed: client connection to 127.15.18.124:44447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:55:49.209652 22324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 02:55:49.211968 22332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:49.212493 22143 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:49.212949 22143 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.005s sys 0.009s
I20260929 02:55:49.213111 22143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal
uuid: "5fff2b419dcf43688659f1bdb5541007"
format_stamp: "Formatted at 2026-09-29 02:55:48 on dist-test-slave-q5f0"
server_key: "f4689debb803435bf695145c55a8b320"
server_key_iv: "6b78b2d8ba9f801e95138890a39608d3"
server_key_version: "encryptionkey@0"
I20260929 02:55:49.212764 22327 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 943a13ef3a5045c582fa14727c05da9f (127.15.18.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.213518 22189 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 943a13ef3a5045c582fa14727c05da9f (127.15.18.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.217303 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.217584 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:55:49.218025 22139 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.218152 22328 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.218372 22139 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:49.218391 22328 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:49.218604 22143 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:49.225981 22143 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:55:49.226210 22143 log.cc:1196] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 02:55:49.226703 22143 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: Deleting consensus metadata
I20260929 02:55:49.226994 22143 master_runner.cc:315] Copying system tablet from 127.15.18.126:43925
I20260929 02:55:49.229732 22143 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: tablet copy: Beginning tablet copy session from remote peer at address 127.15.18.126:43925
I20260929 02:55:49.245172 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.230626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.15.18.126:43925)
0929 02:55:49.230852 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.230859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.231380 (+ 521us) server_negotiation.cc:407] Connection header received
0929 02:55:49.231581 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.231587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.231671 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.231772 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.232790 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.233915 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.235776 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.236464 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.236586 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.238898 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.238905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.238911 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.238951 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.242348 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.243118 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.243124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.243128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.243290 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.243711 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.243716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.243719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.244056 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.244200 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.244600 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.244823 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 02:55:49.245311 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.230579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:59598)
0929 02:55:49.231248 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.231267 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.231427 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.231819 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.231825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.232349 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.232632 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.232644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.234065 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.234075 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.235629 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.235646 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.236626 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.236632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.236837 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.237592 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.237615 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.238691 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.242502 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.242507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.242512 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.242997 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.243414 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.243419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.243423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.243613 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.244175 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.244180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.244362 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.244514 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.244849 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":382,"threads_started":1}
I20260929 02:55:49.246194 21601 tablet_copy_service.cc:140] P f3079672211e419b8cf87e10fb5134a5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5fff2b419dcf43688659f1bdb5541007 ({username='oryx', principal='oryx/127.15.18.124@KRBTEST.COM'} at 127.0.0.1:59598)
I20260929 02:55:49.246369 21601 tablet_copy_service.cc:161] P f3079672211e419b8cf87e10fb5134a5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5fff2b419dcf43688659f1bdb5541007 at {username='oryx', principal='oryx/127.15.18.124@KRBTEST.COM'} at 127.0.0.1:59598: session id = 5fff2b419dcf43688659f1bdb5541007-00000000000000000000000000000000
I20260929 02:55:49.250072 21601 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:55:49.252125 22143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:55:49.258756 22143 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: tablet copy: Starting download of 0 data blocks...
I20260929 02:55:49.259150 22143 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: tablet copy: Starting download of 1 WAL segments...
I20260929 02:55:49.261685 22143 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:55:49.266033 21601 tablet_copy_service.cc:342] P f3079672211e419b8cf87e10fb5134a5: Request end of tablet copy session 5fff2b419dcf43688659f1bdb5541007-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.18.124@KRBTEST.COM'} at 127.0.0.1:59598
I20260929 02:55:49.266165 21601 tablet_copy_service.cc:433] P f3079672211e419b8cf87e10fb5134a5: ending tablet copy session 5fff2b419dcf43688659f1bdb5541007-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5fff2b419dcf43688659f1bdb5541007
I20260929 02:55:49.279345 22143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:49.282016 22339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:49.282500 22340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:49.285604 22143 server_base.cc:1034] running on GCE node
W20260929 02:55:49.285671 22342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:49.286194 22143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:49.286832 22143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:49.288065 22143 hybrid_clock.cc:648] HybridClock initialized: now 1790650549288021 us; error 36 us; skew 500 ppm
I20260929 02:55:49.288334 22143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650549, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:49.290730 22143 init.cc:377] Logged in from keytab as oryx/127.15.18.124@KRBTEST.COM (short username oryx)
I20260929 02:55:49.292685 22143 webserver.cc:460] Webserver started at http://127.15.18.124:43623/ using document root <none> and password file <none>
I20260929 02:55:49.293069 22143 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:49.293135 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 21642
I20260929 02:55:49.293202 22143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:49.296062 22143 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:49.297591 22349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:49.297943 22143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 02:55:49.298063 22143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal
uuid: "5fff2b419dcf43688659f1bdb5541007"
format_stamp: "Formatted at 2026-09-29 02:55:48 on dist-test-slave-q5f0"
server_key: "f4689debb803435bf695145c55a8b320"
server_key_iv: "6b78b2d8ba9f801e95138890a39608d3"
server_key_version: "encryptionkey@0"
I20260929 02:55:49.298271 22143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:49.314556 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:40765
--local_ip_for_outbound_sockets=127.15.18.65
--tserver_master_addrs=127.15.18.126:43925
--webserver_port=39913
--webserver_interface=127.15.18.65
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab
--principal=oryx/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.18.126:43925,127.15.18.125:33733
--tserver_master_addrs=127.15.18.126:43925,127.15.18.125:33733,127.15.18.124:44447 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:55:49.314826 22143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:49.318168 22143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:49.340808 22143 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.124:44447
I20260929 02:55:49.340847 22401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.124:44447 every 8 connection(s)
I20260929 02:55:49.341920 22143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
I20260929 02:55:49.347131 22402 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:55:49.350929 22402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: Bootstrap starting.
I20260929 02:55:49.368903 22402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:55:49.369387 22402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: Bootstrap complete.
I20260929 02:55:49.370499 22402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } }
I20260929 02:55:49.370803 22402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5fff2b419dcf43688659f1bdb5541007, State: Initialized, Role: LEARNER
I20260929 02:55:49.371107 22402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } }
I20260929 02:55:49.371958 22405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.372249 22402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:49.372364 22405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:55:49.389897 22416 catalog_manager.cc:1259] Loaded cluster ID: fb387822bc0a47e7ab62ee0e6ba510b7
I20260929 02:55:49.390002 22416 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: loading cluster ID for follower catalog manager: success
I20260929 02:55:49.392993 22416 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: acquiring CA information for follower catalog manager: success
I20260929 02:55:49.394953 22416 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:55:49.690759 22356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:49.691035 22356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:49.691124 22356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:49.698349 22356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:49.698479 22356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:49.698601 22356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:49.708336 22356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab
--principal=oryx/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:40765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=39913
--tserver_master_addrs=127.15.18.126:43925,127.15.18.125:33733,127.15.18.124:44447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:49.709714 22356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:49.710996 22356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 02:55:49.717346 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.710023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.15.18.124:44447)
0929 02:55:49.711990 (+ 1967us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.711994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.712012 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:49.712070 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.712073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.712139 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.712240 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:49.712623 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.713554 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.715607 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.716446 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.716560 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.716708 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.716999 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.717104 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1863,"thread_start_us":1748,"threads_started":1}
I20260929 02:55:49.717427 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.709915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:44447 (local address 127.0.0.1:51102)
0929 02:55:49.711711 (+ 1796us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.711733 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.711877 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.712284 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.712288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.712315 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:49.712495 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.712506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.713709 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.713715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.715447 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.715463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.716630 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.716636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.716818 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.716862 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.716976 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.717116 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1663,"thread_start_us":1545,"threads_started":1}
I20260929 02:55:49.718418 22377 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 02:55:49.719267 22405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3079672211e419b8cf87e10fb5134a5. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:49.719460 22405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: This master's current role is: LEARNER
W20260929 02:55:49.725447 22423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:49.725557 22426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:49.725452 22424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:49.726271 22356 server_base.cc:1034] running on GCE node
I20260929 02:55:49.726886 22356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:49.727849 22356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:49.729107 22356 hybrid_clock.cc:648] HybridClock initialized: now 1790650549729055 us; error 43 us; skew 500 ppm
I20260929 02:55:49.729451 22356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650549, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:49.732429 22356 init.cc:377] Logged in from keytab as oryx/127.15.18.65@KRBTEST.COM (short username oryx)
I20260929 02:55:49.733824 22356 webserver.cc:460] Webserver started at http://127.15.18.65:39913/ using document root <none> and password file <none>
I20260929 02:55:49.734477 22356 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:49.734599 22356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:49.739851 22356 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:55:49.742568 22433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:49.744092 22356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:49.744258 22356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "0e0628f790c74c0bad86b970bbf2bd80"
format_stamp: "Formatted at 2026-09-29 02:55:46 on dist-test-slave-q5f0"
server_key: "f62d0b1fccebeca2d5c04e809d5a5ae5"
server_key_iv: "909038b7ddc13f7e1561cc7ac77944c2"
server_key_version: "encryptionkey@0"
I20260929 02:55:49.745201 22356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:49.767630 22356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:49.772562 22356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:49.772855 22356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:49.773829 22356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:49.775175 22356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:49.775270 22356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:49.775374 22356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:49.775440 22356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:49.809104 22356 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:40765
I20260929 02:55:49.809168 22545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:40765 every 8 connection(s)
I20260929 02:55:49.810762 22356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:49.811359 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 22356
I20260929 02:55:49.811690 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 21776
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.66@KRBTEST.COM
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650549, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.124@KRBTEST.COM
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650549, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:49.857189 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.812642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.15.18.66:45497)
0929 02:55:49.828223 (+ 15581us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.828235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.830860 (+ 2625us) server_negotiation.cc:407] Connection header received
0929 02:55:49.831008 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.831019 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.831315 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.831467 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.833038 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.834061 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.835702 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.836396 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.836522 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.845096 (+ 8574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.845116 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.845135 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.845181 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.851441 (+ 6260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.853547 (+ 2106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.853559 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.853562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.853724 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.854195 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.854200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.854204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.855284 (+ 1080us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.855658 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.856314 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.856613 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15368,"thread_start_us":15598,"threads_started":1}
I20260929 02:55:49.858713 21899 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 29 02:55:49 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650549, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.125@KRBTEST.COM
I20260929 02:55:49.859594 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.813776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:36761 (local address 127.15.18.124:44447)
0929 02:55:49.813996 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.814003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.815849 (+ 1846us) server_negotiation.cc:407] Connection header received
0929 02:55:49.838913 (+ 23064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.838921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.839018 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.839121 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.841712 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.842797 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.845412 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.846066 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.846169 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.851610 (+ 5441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.851617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.851621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.851664 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.854884 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.855845 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.855851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.855855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.856001 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.857197 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.857202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.857205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.857543 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.857668 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.859016 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.859236 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:55:49.862149 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.814530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:44447 (local address 127.15.18.65:36761)
0929 02:55:49.815641 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.815710 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.836361 (+ 20651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.839659 (+ 3298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.839673 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.840504 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.841471 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.841500 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.843025 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.843030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.845227 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.845244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.846575 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.846582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.847199 (+ 617us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.847982 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.848039 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.851302 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.855041 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.855051 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.855070 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.855725 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.856773 (+ 1048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.856778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.856781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.857065 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.858050 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.858058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.858515 (+ 457us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.860088 (+ 1573us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.860707 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":360,"threads_started":1}
I20260929 02:55:49.864027 22546 heartbeater.cc:347] Connected to a master server at 127.15.18.124:44447
I20260929 02:55:49.864398 22546 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:49.865319 22546 heartbeater.cc:511] Master 127.15.18.124:44447 requested a full tablet report, sending...
I20260929 02:55:49.865728 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.818842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:44475 (local address 127.15.18.126:43925)
0929 02:55:49.844505 (+ 25663us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.844512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.845393 (+ 881us) server_negotiation.cc:407] Connection header received
0929 02:55:49.846229 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.846236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.846351 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.846443 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.848245 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.849438 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.851484 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.852225 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.852352 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.856553 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.856559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.856565 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.856606 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.860016 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.861007 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.861014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.861018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.861178 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.862091 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.862097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.862101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.862749 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.864929 (+ 2180us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.865094 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.865316 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25519,"thread_start_us":25671,"threads_started":1}
I20260929 02:55:49.866504 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.818408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.65:44475)
0929 02:55:49.845097 (+ 26689us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.845119 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.845983 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.846658 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.846663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.847616 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.847952 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.847965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.849656 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.849662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.851288 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.851305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.852627 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.852634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.852816 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.853341 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.853364 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.856316 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.860200 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.860205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.860210 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.860851 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.861613 (+ 762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.861619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.861623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.861786 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.863319 (+ 1533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.863324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.863503 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.863661 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.864001 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26546,"thread_start_us":28286,"threads_started":1}
I20260929 02:55:49.867452 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.825058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:33733 (local address 127.15.18.65:56869)
0929 02:55:49.851009 (+ 25951us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:49.851034 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:49.851183 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:49.851825 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:49.851830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:49.852298 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:49.852632 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.852643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.854100 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.854105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.855588 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:49.855602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.856979 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.856984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:49.857176 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.857652 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:49.857673 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:49.860306 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:49.864018 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.864023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.864026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.864595 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.865029 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:49.865033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:49.865037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:49.865191 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:49.866272 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:49.866276 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:49.866442 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 02:55:49.866803 (+ 361us) client_negotiation.cc:240] Negotiation successful
0929 02:55:49.867076 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25820,"thread_start_us":31567,"threads_started":1}
I20260929 02:55:49.867468 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:49.825489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:56869 (local address 127.15.18.125:33733)
0929 02:55:49.850466 (+ 24977us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:49.850472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:49.851310 (+ 838us) server_negotiation.cc:407] Connection header received
0929 02:55:49.851382 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:49.851387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:49.851486 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:49.851610 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:49.852827 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.853925 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.855759 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:49.856371 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:49.856485 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:49.860522 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:49.860529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:49.860533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:49.860576 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:49.863865 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.864733 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.864738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.864741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.864897 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:49.865322 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:49.865326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:49.865329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:49.865745 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:49.866215 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:49.866918 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 02:55:49.867115 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24836,"thread_start_us":24988,"threads_started":1}
I20260929 02:55:49.867986 22367 ts_manager.cc:194] Registered new tserver with Master: 0e0628f790c74c0bad86b970bbf2bd80 (127.15.18.65:40765)
I20260929 02:55:49.868624 22551 heartbeater.cc:347] Connected to a master server at 127.15.18.125:33733
I20260929 02:55:49.868752 22551 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:49.869200 22551 heartbeater.cc:511] Master 127.15.18.125:33733 requested a full tablet report, sending...
I20260929 02:55:49.870191 22547 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43925
I20260929 02:55:49.870314 22547 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:49.870707 22547 heartbeater.cc:511] Master 127.15.18.126:43925 requested a full tablet report, sending...
I20260929 02:55:49.870918 22179 ts_manager.cc:194] Registered new tserver with Master: 0e0628f790c74c0bad86b970bbf2bd80 (127.15.18.65:40765)
I20260929 02:55:49.871814 21581 ts_manager.cc:194] Re-registered known tserver with Master: 0e0628f790c74c0bad86b970bbf2bd80 (127.15.18.65:40765)
I20260929 02:55:49.872598 21581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:44475
I20260929 02:55:49.948484 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 21776
I20260929 02:55:49.961403 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:45497
--local_ip_for_outbound_sockets=127.15.18.66
--tserver_master_addrs=127.15.18.126:43925
--webserver_port=41505
--webserver_interface=127.15.18.66
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab
--principal=oryx/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.18.126:43925,127.15.18.125:33733
--tserver_master_addrs=127.15.18.126:43925,127.15.18.125:33733,127.15.18.124:44447 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:50.283408 22558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:50.283784 22558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:50.283947 22558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:50.291342 22558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:50.291474 22558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:50.291641 22558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:50.301959 22558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab
--principal=oryx/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:45497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=41505
--tserver_master_addrs=127.15.18.126:43925,127.15.18.125:33733,127.15.18.124:44447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:50.303293 22558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:50.304728 22558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:50.318485 22563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:50.318435 22564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:50.318423 22566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:50.319185 22558 server_base.cc:1034] running on GCE node
I20260929 02:55:50.319845 22558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:50.320827 22558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:50.322064 22558 hybrid_clock.cc:648] HybridClock initialized: now 1790650550322006 us; error 48 us; skew 500 ppm
I20260929 02:55:50.322365 22558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:50 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650550, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:50.325469 22558 init.cc:377] Logged in from keytab as oryx/127.15.18.66@KRBTEST.COM (short username oryx)
I20260929 02:55:50.326917 22558 webserver.cc:460] Webserver started at http://127.15.18.66:41505/ using document root <none> and password file <none>
I20260929 02:55:50.327620 22558 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:50.327751 22558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:50.333179 22558 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:55:50.333417 22328 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: NON_VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: true } } }
I20260929 02:55:50.333729 22328 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:50.335038 22328 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5: attempting to promote NON_VOTER 5fff2b419dcf43688659f1bdb5541007 to VOTER
I20260929 02:55:50.335745 22328 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } }
I20260929 02:55:50.336345 22576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:50.337636 22377 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 LEARNER]: Refusing update from remote peer f3079672211e419b8cf87e10fb5134a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 02:55:50.338050 22558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:50.338053 22189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Refusing update from remote peer f3079672211e419b8cf87e10fb5134a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 02:55:50.338137 22328 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 02:55:50.338591 22574 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 02:55:50.338217 22558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "a4258656104f48bfbebd1b16b87233d0"
format_stamp: "Formatted at 2026-09-29 02:55:47 on dist-test-slave-q5f0"
server_key: "8ae1f19bcf2b08e4bd6bcc90e62064bd"
server_key_iv: "6f2a72233917d3d0f5f0ada5e0a24045"
server_key_version: "encryptionkey@0"
I20260929 02:55:50.339569 22558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:50.357555 22574 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5fff2b419dcf43688659f1bdb5541007 (127.15.18.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:50.360634 22188 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5fff2b419dcf43688659f1bdb5541007 (127.15.18.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:50.360834 22377 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5fff2b419dcf43688659f1bdb5541007 (127.15.18.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:50.364405 22584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:50.364701 22584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:55:50.366953 22585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:50.367300 22585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:55:50.367792 22573 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:50.368096 22573 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:50.368371 22574 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:50.368590 22574 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:50.368511 22573 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3079672211e419b8cf87e10fb5134a5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:50.368729 22573 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5 [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:50.385748 22558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:50.390424 22558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:50.390717 22558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:50.391813 22558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:50.393220 22558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:50.393313 22558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:50.393430 22558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:50.393512 22558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:50.423756 22558 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:45497
I20260929 02:55:50.423825 22694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:45497 every 8 connection(s)
I20260929 02:55:50.425175 22558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:50.431450 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 22558
I20260929 02:55:50.431751 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 21909
Sep 29 02:55:50 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650550, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.124@KRBTEST.COM
Sep 29 02:55:50 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650550, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
Sep 29 02:55:50 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650550, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.125@KRBTEST.COM
Sep 29 02:55:50 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.67@KRBTEST.COM
I20260929 02:55:50.476279 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:50.428709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.66:41083)
0929 02:55:50.452166 (+ 23457us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:50.452189 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:50.452333 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:50.458411 (+ 6078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:50.458416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:50.459060 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:50.459402 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:50.459415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.461676 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.461681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:50.463804 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:50.463817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.464983 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.464989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:50.465166 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:50.465736 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:50.465756 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:50.468436 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:50.472468 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:50.472473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:50.472477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:50.473042 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:50.473500 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:50.473505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:50.473508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:50.473645 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:50.474352 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:50.474357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:50.474510 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 02:55:50.474645 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 02:55:50.474918 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23259,"mutex_wait_us":32,"spinlock_wait_cycles":19328,"thread_start_us":14642,"threads_started":1}
I20260929 02:55:50.476306 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:50.428489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:41083 (local address 127.15.18.126:43925)
0929 02:55:50.458036 (+ 29547us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:50.458042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:50.458060 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:50.458118 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:50.458123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:50.458215 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:50.459577 (+ 1362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:50.459742 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.460753 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.463986 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.464592 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.464710 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:50.468652 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:50.468659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:50.468664 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:50.468716 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:50.472292 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:50.473202 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:50.473207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:50.473210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:50.473367 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:50.473791 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:50.473797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:50.473800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:50.474133 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:50.474302 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:50.475742 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0929 02:55:50.475968 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29429,"thread_start_us":29288,"threads_started":1}
I20260929 02:55:50.476806 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:50.429263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:45279 (local address 127.15.18.124:44447)
0929 02:55:50.452824 (+ 23561us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:50.452831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:50.452852 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:55:50.452931 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:50.452937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:50.453019 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:50.453870 (+ 851us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:50.456328 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.457456 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.460304 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.460992 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.461102 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:50.465837 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:50.465843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:50.465848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:50.465885 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:50.470959 (+ 5074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:50.471917 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:50.471923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:50.471927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:50.472078 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:50.472667 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:50.472672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:50.472676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:50.473026 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:50.473174 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:50.476177 (+ 3003us) server_negotiation.cc:299] Negotiation successful
0929 02:55:50.476450 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23420,"thread_start_us":23273,"threads_started":1}
I20260929 02:55:50.477102 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:50.428709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:44447 (local address 127.15.18.66:45279)
0929 02:55:50.443313 (+ 14604us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:50.443381 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:50.447338 (+ 3957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:50.454191 (+ 6853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:50.454205 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:50.455064 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:50.456062 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:50.456089 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.457662 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.457669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:50.460090 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:50.460106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.461244 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.461252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:50.461461 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:50.462322 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:50.462360 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:50.465549 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:50.471085 (+ 5536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:50.471096 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:50.471117 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:50.471755 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:50.472215 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:50.472220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:50.472224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:50.472526 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:50.473251 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:50.473262 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:50.473775 (+ 513us) client_negotiation.cc:769] Sending connection context
0929 02:55:50.474089 (+ 314us) client_negotiation.cc:240] Negotiation successful
0929 02:55:50.474727 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14182,"thread_start_us":24438,"threads_started":1}
I20260929 02:55:50.478456 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:50.429647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:33733 (local address 127.15.18.66:37355)
0929 02:55:50.453588 (+ 23941us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:50.453607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:50.453717 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:50.461533 (+ 7816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:50.461538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:50.462076 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:50.462434 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:50.462446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.463802 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.463807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:50.465429 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:50.465444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.466619 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.466625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:50.466814 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:50.467358 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:50.467379 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:50.469890 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:50.474316 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:50.474321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:50.474324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:50.474852 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:50.475265 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:50.475270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:50.475274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:50.475457 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:50.477175 (+ 1718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:50.477180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:50.477577 (+ 397us) client_negotiation.cc:769] Sending connection context
0929 02:55:50.477725 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 02:55:50.478015 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23816,"thread_start_us":22152,"threads_started":1}
I20260929 02:55:50.478603 22696 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43925
I20260929 02:55:50.478675 22695 heartbeater.cc:347] Connected to a master server at 127.15.18.124:44447
I20260929 02:55:50.479031 22695 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:50.479032 22696 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:50.479370 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:50.428209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:37355 (local address 127.15.18.125:33733)
0929 02:55:50.461206 (+ 32997us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:50.461212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:50.461230 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:50.461287 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:50.461292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:50.461376 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:50.461487 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:50.462585 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.463651 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.465618 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.466219 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.466336 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:50.470053 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:50.470059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:50.470063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:50.470103 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:50.474166 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:50.474985 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:50.474991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:50.474994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:50.475139 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:50.476523 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:50.476527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:50.476531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:50.476893 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:50.477107 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:50.478704 (+ 1597us) server_negotiation.cc:299] Negotiation successful
0929 02:55:50.478982 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32879,"thread_start_us":32745,"threads_started":1}
I20260929 02:55:50.479983 22695 heartbeater.cc:511] Master 127.15.18.124:44447 requested a full tablet report, sending...
I20260929 02:55:50.480106 22697 heartbeater.cc:347] Connected to a master server at 127.15.18.125:33733
I20260929 02:55:50.480218 22697 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:50.480712 22697 heartbeater.cc:511] Master 127.15.18.125:33733 requested a full tablet report, sending...
I20260929 02:55:50.481021 22696 heartbeater.cc:511] Master 127.15.18.126:43925 requested a full tablet report, sending...
I20260929 02:55:50.481802 22179 ts_manager.cc:194] Registered new tserver with Master: a4258656104f48bfbebd1b16b87233d0 (127.15.18.66:45497)
I20260929 02:55:50.481803 22367 ts_manager.cc:194] Registered new tserver with Master: a4258656104f48bfbebd1b16b87233d0 (127.15.18.66:45497)
I20260929 02:55:50.482653 21581 ts_manager.cc:194] Re-registered known tserver with Master: a4258656104f48bfbebd1b16b87233d0 (127.15.18.66:45497)
I20260929 02:55:50.483767 21581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:41083
I20260929 02:55:50.484375 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:50.432987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.15.18.67:45099)
0929 02:55:50.457063 (+ 24076us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:50.457070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:50.457092 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:55:50.457143 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:50.457149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:50.457345 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:50.457480 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:50.459570 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.461070 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.463225 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:50.464119 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:50.464237 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:50.471080 (+ 6843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:50.471098 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:50.471118 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:50.471160 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:50.475091 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:50.476025 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:50.476034 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:50.476038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:50.476200 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:50.482425 (+ 6225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:50.482431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:50.482435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:50.482980 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:50.483281 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:50.483759 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 02:55:50.484036 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23936,"thread_start_us":23792,"threads_started":1}
I20260929 02:55:50.485636 22032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:50.566977 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 21909
I20260929 02:55:50.579394 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:45099
--local_ip_for_outbound_sockets=127.15.18.67
--tserver_master_addrs=127.15.18.126:43925
--webserver_port=38249
--webserver_interface=127.15.18.67
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab
--principal=oryx/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.18.126:43925,127.15.18.125:33733
--tserver_master_addrs=127.15.18.126:43925,127.15.18.125:33733,127.15.18.124:44447 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:55:50.875204 22547 heartbeater.cc:503] Master 127.15.18.126:43925 was elected leader, sending a full tablet report...
W20260929 02:55:50.902165 22708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:50.902446 22708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:50.902590 22708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:50.910171 22708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:50.910303 22708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:50.910456 22708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:50.920626 22708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab
--principal=oryx/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:45099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=38249
--tserver_master_addrs=127.15.18.126:43925,127.15.18.125:33733,127.15.18.124:44447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:50.921939 22708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:50.923208 22708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:50.936375 22714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:50.936329 22715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:50.936349 22717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:50.941159 22708 server_base.cc:1034] running on GCE node
I20260929 02:55:50.941843 22708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:50.942883 22708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:50.944077 22708 hybrid_clock.cc:648] HybridClock initialized: now 1790650550944030 us; error 49 us; skew 500 ppm
I20260929 02:55:50.944391 22708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:50 dist-test-slave-q5f0 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650550, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:50.947628 22708 init.cc:377] Logged in from keytab as oryx/127.15.18.67@KRBTEST.COM (short username oryx)
I20260929 02:55:50.949173 22708 webserver.cc:460] Webserver started at http://127.15.18.67:38249/ using document root <none> and password file <none>
I20260929 02:55:50.949915 22708 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:50.950030 22708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:50.955415 22708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:50.958432 22724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:50.959776 22708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:55:50.959930 22708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "c45d1de6e9294aee87b262d36df0307f"
format_stamp: "Formatted at 2026-09-29 02:55:47 on dist-test-slave-q5f0"
server_key: "b71f6dc15114b59a67c209122a165764"
server_key_iv: "3b39679c6346744e757d9b60ef9d68fb"
server_key_version: "encryptionkey@0"
I20260929 02:55:50.960880 22708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:50.980865 22708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:50.985119 22708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:50.985378 22708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:50.986317 22708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:55:50.987668 22708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:50.987758 22708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:50.987861 22708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:50.987926 22708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:51.018337 22708 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:45099
I20260929 02:55:51.018374 22836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:45099 every 8 connection(s)
I20260929 02:55:51.019779 22708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:51.028973 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 22708
Sep 29 02:55:51 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650550, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.125@KRBTEST.COM
Sep 29 02:55:51 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650550, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.124@KRBTEST.COM
Sep 29 02:55:51 dist-test-slave-q5f0 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650550, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:51.062311 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.022749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:59475 (local address 127.15.18.125:33733)
0929 02:55:51.040974 (+ 18225us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.040981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.041003 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:55:51.041072 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.041078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.041164 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.041279 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:51.043831 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.044868 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.047326 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.048007 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.048123 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.053734 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:51.053740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:51.053745 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:51.053803 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:51.057276 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.058167 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.058173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.058177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.058339 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.059144 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.059149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.059153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.059481 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:51.059639 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.061684 (+ 2045us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.061913 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18087,"thread_start_us":17992,"threads_started":1}
I20260929 02:55:51.065064 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.023231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:33733 (local address 127.15.18.67:59475)
0929 02:55:51.033674 (+ 10443us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:51.033749 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:51.039584 (+ 5835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:51.041777 (+ 2193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:51.041789 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:51.042686 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:51.043625 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.043650 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.045047 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.045052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.047161 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.047175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.048432 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.048439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.048711 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.049463 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:51.049492 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:51.052485 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:51.057443 (+ 4958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:51.057452 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:51.057471 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:51.058032 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:51.058459 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:51.058464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:51.058468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:51.059008 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:51.059713 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:51.059722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:51.060188 (+ 466us) client_negotiation.cc:769] Sending connection context
0929 02:55:51.060499 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 02:55:51.061045 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10065,"thread_start_us":19351,"threads_started":1}
I20260929 02:55:51.067071 22839 heartbeater.cc:347] Connected to a master server at 127.15.18.125:33733
I20260929 02:55:51.067446 22839 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:51.068466 22839 heartbeater.cc:511] Master 127.15.18.125:33733 requested a full tablet report, sending...
I20260929 02:55:51.070415 22179 ts_manager.cc:194] Registered new tserver with Master: c45d1de6e9294aee87b262d36df0307f (127.15.18.67:45099)
I20260929 02:55:51.071719 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.023585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:44447 (local address 127.15.18.67:46331)
0929 02:55:51.035929 (+ 12344us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:51.035949 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:51.039602 (+ 3653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:51.043431 (+ 3829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:51.043437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:51.044045 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:51.044436 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.044448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.046133 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.046138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.047587 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.047599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.053524 (+ 5925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.053530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.053786 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.054389 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:51.054410 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:51.057370 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:51.067277 (+ 9907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:51.067282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:51.067286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:51.067937 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:51.069797 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:51.069802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:51.069806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:51.069964 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:51.070660 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:51.070665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:51.070828 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 02:55:51.070953 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 02:55:51.071246 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12209,"mutex_wait_us":20,"thread_start_us":9573,"threads_started":1}
I20260929 02:55:51.073185 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.022565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:46331 (local address 127.15.18.124:44447)
0929 02:55:51.042281 (+ 19716us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.042288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.042311 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:55:51.042375 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.042381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.042473 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.044691 (+ 2218us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:51.044848 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.045976 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.052735 (+ 6759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.053358 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.053458 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.063156 (+ 9698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:51.063163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:51.063168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:51.063206 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:51.067164 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.069497 (+ 2333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.069503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.069506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.069665 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.070116 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.070120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.070124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.070519 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:51.072056 (+ 1537us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.072604 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.072827 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19572,"thread_start_us":20658,"threads_started":1}
I20260929 02:55:51.074025 22837 heartbeater.cc:347] Connected to a master server at 127.15.18.124:44447
I20260929 02:55:51.074162 22837 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:51.074615 22837 heartbeater.cc:511] Master 127.15.18.124:44447 requested a full tablet report, sending...
I20260929 02:55:51.075826 22367 ts_manager.cc:194] Registered new tserver with Master: c45d1de6e9294aee87b262d36df0307f (127.15.18.67:45099)
I20260929 02:55:51.076570 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.023231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.67:34041)
0929 02:55:51.043012 (+ 19781us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:51.043032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:51.043151 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:51.044155 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:51.044161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:51.045396 (+ 1235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:51.045830 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.045848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.053523 (+ 7675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.053530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.055162 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.055177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.058228 (+ 3051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.058234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.058419 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.059284 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:51.059305 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:51.063622 (+ 4317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:51.071558 (+ 7936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:51.071563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:51.071568 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:51.072093 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:51.074634 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:51.074640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:51.074644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:51.074857 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:51.075564 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:51.075569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:51.075718 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 02:55:51.075859 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 02:55:51.076148 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19622,"mutex_wait_us":540,"thread_start_us":11824,"threads_started":1}
I20260929 02:55:51.077510 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.023027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:34041 (local address 127.15.18.126:43925)
0929 02:55:51.040618 (+ 17591us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.040625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.043131 (+ 2506us) server_negotiation.cc:407] Connection header received
0929 02:55:51.043269 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.043274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.043367 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.043471 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:51.046020 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.049645 (+ 3625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.057177 (+ 7532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.057837 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.057929 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.065018 (+ 7089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:51.065026 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:51.065031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:51.065078 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:51.069303 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.072483 (+ 3180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.072489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.072492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.074499 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.074991 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.074996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.074999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.075333 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:51.075477 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.076893 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.077146 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17454,"thread_start_us":17355,"threads_started":1}
I20260929 02:55:51.078279 22838 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43925
I20260929 02:55:51.078419 22838 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:51.078812 22838 heartbeater.cc:511] Master 127.15.18.126:43925 requested a full tablet report, sending...
I20260929 02:55:51.079850 21581 ts_manager.cc:194] Re-registered known tserver with Master: c45d1de6e9294aee87b262d36df0307f (127.15.18.67:45099)
I20260929 02:55:51.080941 21581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:34041
I20260929 02:55:51.089517 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.047337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.15.18.125:33733)
0929 02:55:51.062768 (+ 15431us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.062773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.072855 (+ 10082us) server_negotiation.cc:407] Connection header received
0929 02:55:51.073094 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.073099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.073194 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.073276 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:51.075501 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.076577 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.078519 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.079203 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.079292 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.083470 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:51.083477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:51.083481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:51.083545 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:51.086547 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.087303 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.087308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.087311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.087457 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.087843 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.087849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.087852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.088242 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:51.088469 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.088964 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.089178 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15326,"thread_start_us":16661,"threads_started":1}
I20260929 02:55:51.089566 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.046798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.15.18.126:43925)
0929 02:55:51.069472 (+ 22674us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.069478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.069495 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:55:51.069544 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.069549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.069634 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.069746 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:51.072947 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.074252 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.078053 (+ 3801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.078672 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.078782 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.083056 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:51.083063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:51.083068 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:51.083106 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:51.086299 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.087176 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.087182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.087186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.087348 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.087899 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.087904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.087907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.088223 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:51.088469 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.088924 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.089151 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22522,"thread_start_us":25880,"threads_started":1}
I20260929 02:55:51.090390 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.048525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.15.18.124:44447)
0929 02:55:51.063164 (+ 14639us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.063169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.064799 (+ 1630us) server_negotiation.cc:407] Connection header received
0929 02:55:51.068186 (+ 3387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.068192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.068280 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.068380 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:51.073139 (+ 4759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.074616 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.077893 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.079370 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.079476 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.084266 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:51.084273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:51.084278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:51.084315 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:51.086962 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.087615 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.087620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.087624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.087794 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.088826 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.088831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.088834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.089154 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:51.089411 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.089938 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.090143 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14525,"thread_start_us":14381,"threads_started":1}
I20260929 02:55:51.093736 21581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:51.100769 21581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:51.131390 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.117905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:45497 (local address 127.0.0.1:34110)
0929 02:55:51.118505 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:51.118526 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:51.118662 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:51.120480 (+ 1818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:51.120486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:51.120542 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:51.120948 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.120960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.126011 (+ 5051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.126018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.128345 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.128363 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.130598 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.130604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.130793 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.130832 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:55:51.130900 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 02:55:51.131035 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":280,"threads_started":1}
I20260929 02:55:51.133024 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.118231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34110 (local address 127.15.18.66:45497)
0929 02:55:51.119784 (+ 1553us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.119794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.119826 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:55:51.119946 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.119955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.120240 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.120470 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:51.123653 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.125820 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.128588 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.130323 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.132267 (+ 1944us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.132507 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.132584 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.132750 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329,"thread_start_us":1163,"threads_started":1}
I20260929 02:55:51.137255 22630 tablet_service.cc:1467] Processing CreateTablet for tablet 80bbd4fe28b841e8998f8be27d93c5a2 (DEFAULT_TABLE table=test-table [id=54175c18f63b45b7850ed94ce6b8281b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:51.139328 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.118284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:45099 (local address 127.0.0.1:47804)
0929 02:55:51.118813 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:51.118828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:51.118921 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:51.132215 (+ 13294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:51.132219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:51.132245 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:51.132570 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.132581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.134692 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.134698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.136720 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.136736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.138529 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.138535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.138729 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.138764 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:55:51.138838 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:55:51.138974 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":724,"threads_started":1}
I20260929 02:55:51.140179 22630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80bbd4fe28b841e8998f8be27d93c5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:51.140364 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.118850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.15.18.67:45099)
0929 02:55:51.131308 (+ 12458us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.131317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.131353 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:55:51.131457 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.131465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.131800 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.132757 (+ 957us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:51.133039 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.134541 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.136939 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.138323 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.139608 (+ 1285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.139835 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.139902 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.140059 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12276,"thread_start_us":13095,"threads_started":1}
I20260929 02:55:51.147616 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.120720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:40765 (local address 127.0.0.1:40724)
0929 02:55:51.131921 (+ 11201us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:51.131938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:51.132024 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:51.141276 (+ 9252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:51.141282 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:51.141313 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:51.141480 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.141491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.143248 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.143255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.145152 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.145167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.146826 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.146833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.147015 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.147048 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:55:51.147147 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:55:51.147285 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11084,"thread_start_us":23078,"threads_started":1}
I20260929 02:55:51.148000 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.120834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.15.18.65:40765)
0929 02:55:51.140576 (+ 19742us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.140586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.140619 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:55:51.140727 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.140736 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.141036 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.141256 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:51.141657 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.143078 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.145332 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.146645 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.146783 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.146990 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.147194 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.147345 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19543,"thread_start_us":20602,"threads_started":1}
I20260929 02:55:51.148065 22772 tablet_service.cc:1467] Processing CreateTablet for tablet 80bbd4fe28b841e8998f8be27d93c5a2 (DEFAULT_TABLE table=test-table [id=54175c18f63b45b7850ed94ce6b8281b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:51.149459 22772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80bbd4fe28b841e8998f8be27d93c5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:51.153966 22871 tablet_bootstrap.cc:492] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: Bootstrap starting.
I20260929 02:55:51.155308 22481 tablet_service.cc:1467] Processing CreateTablet for tablet 80bbd4fe28b841e8998f8be27d93c5a2 (DEFAULT_TABLE table=test-table [id=54175c18f63b45b7850ed94ce6b8281b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:51.156772 22481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80bbd4fe28b841e8998f8be27d93c5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:51.157058 22871 tablet_bootstrap.cc:654] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:51.157992 22871 log.cc:824] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:51.160436 22871 tablet_bootstrap.cc:492] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: No bootstrap required, opened a new log
I20260929 02:55:51.160714 22871 ts_tablet_manager.cc:1397] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 02:55:51.164508 22873 tablet_bootstrap.cc:492] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: Bootstrap starting.
I20260929 02:55:51.168102 22873 tablet_bootstrap.cc:654] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:51.169302 22873 log.cc:824] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:51.169654 22871 raft_consensus.cc:348] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:51.170125 22871 raft_consensus.cc:374] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:51.170240 22871 raft_consensus.cc:729] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4258656104f48bfbebd1b16b87233d0, State: Initialized, Role: FOLLOWER
I20260929 02:55:51.170266 22874 tablet_bootstrap.cc:492] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: Bootstrap starting.
I20260929 02:55:51.170897 22871 consensus_queue.cc:260] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:51.172646 22696 heartbeater.cc:503] Master 127.15.18.126:43925 was elected leader, sending a full tablet report...
I20260929 02:55:51.173791 22874 tablet_bootstrap.cc:654] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:51.174619 22871 ts_tablet_manager.cc:1428] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260929 02:55:51.175048 22874 log.cc:824] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:51.178851 22874 tablet_bootstrap.cc:492] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: No bootstrap required, opened a new log
I20260929 02:55:51.178896 22873 tablet_bootstrap.cc:492] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: No bootstrap required, opened a new log
I20260929 02:55:51.179128 22873 ts_tablet_manager.cc:1397] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260929 02:55:51.179129 22874 ts_tablet_manager.cc:1397] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260929 02:55:51.189363 22873 raft_consensus.cc:348] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:51.189364 22874 raft_consensus.cc:348] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:51.189846 22874 raft_consensus.cc:374] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:51.189849 22873 raft_consensus.cc:374] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:51.189949 22873 raft_consensus.cc:729] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45d1de6e9294aee87b262d36df0307f, State: Initialized, Role: FOLLOWER
I20260929 02:55:51.189949 22874 raft_consensus.cc:729] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0628f790c74c0bad86b970bbf2bd80, State: Initialized, Role: FOLLOWER
I20260929 02:55:51.190590 22874 consensus_queue.cc:260] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:51.190587 22873 consensus_queue.cc:260] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:51.192407 22874 ts_tablet_manager.cc:1428] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260929 02:55:51.194978 22838 heartbeater.cc:503] Master 127.15.18.126:43925 was elected leader, sending a full tablet report...
I20260929 02:55:51.194988 22873 ts_tablet_manager.cc:1428] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260929 02:55:51.206847 22698 tablet.cc:2369] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:55:51.272608 22840 tablet.cc:2369] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:51.299438 22876 raft_consensus.cc:482] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:51.299746 22876 raft_consensus.cc:504] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:51.301390 22876 leader_election.cc:288] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c45d1de6e9294aee87b262d36df0307f (127.15.18.67:45099), 0e0628f790c74c0bad86b970bbf2bd80 (127.15.18.65:40765)
I20260929 02:55:51.307314 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.301779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:33879 (local address 127.15.18.67:45099)
0929 02:55:51.302004 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.302011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.302433 (+ 422us) server_negotiation.cc:407] Connection header received
0929 02:55:51.302618 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.302622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.302683 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.302762 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:51.303305 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.304059 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.305726 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.306501 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.306642 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.306716 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.306962 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.307077 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:55:51.307370 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.301698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:45099 (local address 127.15.18.66:33879)
0929 02:55:51.302320 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:51.302336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:51.302487 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:51.302794 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:51.302803 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:51.302830 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:51.303167 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.303178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.304181 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.304186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.305580 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.305591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.306689 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.306695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.306825 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.306848 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:55:51.306924 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 02:55:51.307049 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":308,"threads_started":1}
I20260929 02:55:51.308202 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.302534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:57425 (local address 127.15.18.65:40765)
0929 02:55:51.302731 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.302737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.303340 (+ 603us) server_negotiation.cc:407] Connection header received
0929 02:55:51.303477 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.303481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.303572 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.303641 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:51.304090 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.304888 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.306575 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.307308 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.307466 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.307789 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.307840 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.307962 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:55:51.308238 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.302405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:40765 (local address 127.15.18.66:57425)
0929 02:55:51.303221 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:51.303235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:51.303354 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:51.303675 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:51.303679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:51.303700 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:51.303955 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.303963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.305040 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.305045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.306434 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:51.306445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.307546 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.307553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:51.307712 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.307743 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:55:51.307824 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:55:51.307944 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":553,"threads_started":1}
I20260929 02:55:51.308454 22792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80bbd4fe28b841e8998f8be27d93c5a2" candidate_uuid: "a4258656104f48bfbebd1b16b87233d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45d1de6e9294aee87b262d36df0307f" is_pre_election: true
I20260929 02:55:51.309012 22792 raft_consensus.cc:2454] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4258656104f48bfbebd1b16b87233d0 in term 0.
I20260929 02:55:51.309096 22501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80bbd4fe28b841e8998f8be27d93c5a2" candidate_uuid: "a4258656104f48bfbebd1b16b87233d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0628f790c74c0bad86b970bbf2bd80" is_pre_election: true
I20260929 02:55:51.309581 22501 raft_consensus.cc:2454] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4258656104f48bfbebd1b16b87233d0 in term 0.
I20260929 02:55:51.309643 22582 leader_election.cc:302] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4258656104f48bfbebd1b16b87233d0, c45d1de6e9294aee87b262d36df0307f; no voters:
I20260929 02:55:51.310029 22876 raft_consensus.cc:2781] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:51.310156 22876 raft_consensus.cc:482] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:51.310252 22876 raft_consensus.cc:3037] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:51.312414 22876 raft_consensus.cc:504] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:51.312960 22876 leader_election.cc:288] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [CANDIDATE]: Term 1 election: Requested vote from peers c45d1de6e9294aee87b262d36df0307f (127.15.18.67:45099), 0e0628f790c74c0bad86b970bbf2bd80 (127.15.18.65:40765)
I20260929 02:55:51.313485 22792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80bbd4fe28b841e8998f8be27d93c5a2" candidate_uuid: "a4258656104f48bfbebd1b16b87233d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45d1de6e9294aee87b262d36df0307f"
I20260929 02:55:51.313565 22501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80bbd4fe28b841e8998f8be27d93c5a2" candidate_uuid: "a4258656104f48bfbebd1b16b87233d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0628f790c74c0bad86b970bbf2bd80"
I20260929 02:55:51.313747 22792 raft_consensus.cc:3037] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:51.313746 22501 raft_consensus.cc:3037] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:51.317019 22792 raft_consensus.cc:2454] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4258656104f48bfbebd1b16b87233d0 in term 1.
I20260929 02:55:51.317019 22501 raft_consensus.cc:2454] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4258656104f48bfbebd1b16b87233d0 in term 1.
I20260929 02:55:51.317478 22582 leader_election.cc:302] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0628f790c74c0bad86b970bbf2bd80, a4258656104f48bfbebd1b16b87233d0; no voters:
I20260929 02:55:51.317807 22876 raft_consensus.cc:2781] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:51.318241 22876 raft_consensus.cc:686] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 1 LEADER]: Becoming Leader. State: Replica: a4258656104f48bfbebd1b16b87233d0, State: Running, Role: LEADER
I20260929 02:55:51.318789 22876 consensus_queue.cc:237] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:51.324122 21580 catalog_manager.cc:5495] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4258656104f48bfbebd1b16b87233d0 (127.15.18.66). New cstate: current_term: 1 leader_uuid: "a4258656104f48bfbebd1b16b87233d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } health_report { overall_health: UNKNOWN } } }
W20260929 02:55:51.347244 22556 tablet.cc:2369] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:51.369268 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.364379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.15.18.66:45497)
0929 02:55:51.364530 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.364535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.364551 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:55:51.364751 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.364755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.364831 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.364952 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:51.365241 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.366022 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.367301 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.367778 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.367877 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.368314 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:51.368852 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:51.368956 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.368996 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.369058 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 02:55:51.377691 22501 raft_consensus.cc:1261] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80 [term 1 FOLLOWER]: Refusing update from remote peer a4258656104f48bfbebd1b16b87233d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:51.377691 22792 raft_consensus.cc:1261] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f [term 1 FOLLOWER]: Refusing update from remote peer a4258656104f48bfbebd1b16b87233d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:51.378712 22876 consensus_queue.cc:1035] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:51.379060 22882 consensus_queue.cc:1035] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:51.403957 22887 mvcc.cc:205] Tried to move back new op lower bound from 7334504658431987712 to 7334504658205659136. Current Snapshot: MvccSnapshot[applied={T|T < 7334504658431987712}]
I20260929 02:55:51.408473 21581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:51.408918 21581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:51.414851 21581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3079672211e419b8cf87e10fb5134a5: Sending DeleteTablet for 3 replicas of tablet 80bbd4fe28b841e8998f8be27d93c5a2
I20260929 02:55:51.415923 22772 tablet_service.cc:1514] Processing DeleteTablet for tablet 80bbd4fe28b841e8998f8be27d93c5a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:51 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:47804
I20260929 02:55:51.416096 22481 tablet_service.cc:1514] Processing DeleteTablet for tablet 80bbd4fe28b841e8998f8be27d93c5a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:51 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:40724
I20260929 02:55:51.422492 22630 tablet_service.cc:1514] Processing DeleteTablet for tablet 80bbd4fe28b841e8998f8be27d93c5a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:51 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:34110
I20260929 02:55:51.434356 22895 tablet_replica.cc:330] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: stopping tablet replica
I20260929 02:55:51.434863 22895 raft_consensus.cc:2229] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:51.435400 22895 raft_consensus.cc:2258] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:51.436308 22897 tablet_replica.cc:330] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: stopping tablet replica
I20260929 02:55:51.436761 22897 raft_consensus.cc:2229] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:51.437444 22897 raft_consensus.cc:2258] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:51.439126 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 21549
I20260929 02:55:51.440300 21611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:51.440759 22895 ts_tablet_manager.cc:1905] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:51.445511 22897 ts_tablet_manager.cc:1905] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:51.448917 22894 tablet_replica.cc:330] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: stopping tablet replica
I20260929 02:55:51.449301 22894 raft_consensus.cc:2229] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:51.449820 22894 raft_consensus.cc:2258] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:51.452250 22894 ts_tablet_manager.cc:1905] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:51.459852 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.418954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.15.18.125:33733)
0929 02:55:51.419094 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.419100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.437142 (+ 18042us) server_negotiation.cc:407] Connection header received
0929 02:55:51.438893 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.438898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.439003 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.439080 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:51.441868 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.442987 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.445027 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.445637 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.445748 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.453022 (+ 7274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:51.453029 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:51.453034 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:51.453085 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:51.456423 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.457289 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.457294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.457298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.457451 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.457890 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.457895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.457899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.458246 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:51.458499 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.459196 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.459415 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 02:55:51.466243 22895 ts_tablet_manager.cc:1918] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:51.466435 22895 log.cc:1196] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/80bbd4fe28b841e8998f8be27d93c5a2
I20260929 02:55:51.467000 22895 ts_tablet_manager.cc:1939] T 80bbd4fe28b841e8998f8be27d93c5a2 P 0e0628f790c74c0bad86b970bbf2bd80: Deleting consensus metadata
I20260929 02:55:51.467942 22897 ts_tablet_manager.cc:1918] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:51.468129 22897 log.cc:1196] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/80bbd4fe28b841e8998f8be27d93c5a2
I20260929 02:55:51.468694 22897 ts_tablet_manager.cc:1939] T 80bbd4fe28b841e8998f8be27d93c5a2 P a4258656104f48bfbebd1b16b87233d0: Deleting consensus metadata
I20260929 02:55:51.473445 22894 ts_tablet_manager.cc:1918] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:51.473726 22894 log.cc:1196] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/80bbd4fe28b841e8998f8be27d93c5a2
I20260929 02:55:51.474253 22894 ts_tablet_manager.cc:1939] T 80bbd4fe28b841e8998f8be27d93c5a2 P c45d1de6e9294aee87b262d36df0307f: Deleting consensus metadata
I20260929 02:55:51.479106 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.437777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.15.18.124:44447)
0929 02:55:51.438009 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.438016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.462512 (+ 24496us) server_negotiation.cc:407] Connection header received
0929 02:55:51.462748 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.462754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.462836 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.462899 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:51.464861 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.466037 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.468338 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.468958 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.469058 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.472986 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:51.472993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:51.472997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:51.473038 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:51.476477 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.477228 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.477234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.477238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.477366 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.477724 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.477729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.477733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.478016 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:51.478248 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.478575 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.478799 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:55:51.589023 21565 catalog_manager.cc:4841] TS a4258656104f48bfbebd1b16b87233d0 (127.15.18.66:45497): tablet 80bbd4fe28b841e8998f8be27d93c5a2 (table test-table [id=54175c18f63b45b7850ed94ce6b8281b]) successfully deleted
I20260929 02:55:51.589051 21567 catalog_manager.cc:4841] TS 0e0628f790c74c0bad86b970bbf2bd80 (127.15.18.65:40765): tablet 80bbd4fe28b841e8998f8be27d93c5a2 (table test-table [id=54175c18f63b45b7850ed94ce6b8281b]) successfully deleted
I20260929 02:55:51.589439 21567 catalog_manager.cc:4841] TS c45d1de6e9294aee87b262d36df0307f (127.15.18.67:45099): tablet 80bbd4fe28b841e8998f8be27d93c5a2 (table test-table [id=54175c18f63b45b7850ed94ce6b8281b]) successfully deleted
I20260929 02:55:51.589819 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 21549
I20260929 02:55:51.637038 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.611326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.15.18.124:44447)
0929 02:55:51.611664 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.611670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.622085 (+ 10415us) server_negotiation.cc:407] Connection header received
0929 02:55:51.622628 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.622635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.622728 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.622801 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:51.625088 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.626246 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.628218 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.628626 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.628732 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.631833 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:51.631838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:51.631841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:51.631883 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:51.634608 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.635228 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.635232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.635235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.635362 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.635742 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.635746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.635748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.636019 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:51.636226 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.636604 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.636801 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:55:51.642537 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:51.610976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.15.18.125:33733)
0929 02:55:51.611168 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:51.611175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:51.627679 (+ 16504us) server_negotiation.cc:407] Connection header received
0929 02:55:51.627838 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:51.627843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:51.627937 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:51.628034 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:51.630135 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.631231 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.633013 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:51.633569 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:51.633701 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:51.637294 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:51.637300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:51.637304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:51.637343 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:51.640091 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.640833 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.640836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.640839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.640978 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:51.641340 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:51.641343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:51.641345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:51.641616 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:51.641791 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:51.642141 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 02:55:51.642302 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:55:52.400879 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.400174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.65:42427)
0929 02:55:52.400597 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.400717 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":213,"threads_started":1}
W20260929 02:55:52.401332 22547 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:43925 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
I20260929 02:55:52.403712 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.402566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.66:39151)
0929 02:55:52.403378 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.403533 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":589,"threads_started":1}
W20260929 02:55:52.404008 22696 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:43925 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
I20260929 02:55:52.411190 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.410252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.67:57155)
0929 02:55:52.410833 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.411011 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":349,"threads_started":1}
W20260929 02:55:52.411540 22838 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:43925 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
I20260929 02:55:52.915921 22909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3079672211e419b8cf87e10fb5134a5)
I20260929 02:55:52.916204 22909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } }
I20260929 02:55:52.917583 22909 leader_election.cc:288] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3079672211e419b8cf87e10fb5134a5 (127.15.18.126:43925), 5fff2b419dcf43688659f1bdb5541007 (127.15.18.124:44447)
I20260929 02:55:52.918473 22910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3079672211e419b8cf87e10fb5134a5)
I20260929 02:55:52.918705 22910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } }
I20260929 02:55:52.919672 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.917764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:39172)
0929 02:55:52.919242 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.919403 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1340,"mutex_wait_us":450,"thread_start_us":728,"threads_started":1}
W20260929 02:55:52.919958 22164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
I20260929 02:55:52.922801 22910 leader_election.cc:288] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3079672211e419b8cf87e10fb5134a5 (127.15.18.126:43925), 943a13ef3a5045c582fa14727c05da9f (127.15.18.125:33733)
I20260929 02:55:52.925302 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.919666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:39182)
0929 02:55:52.924803 (+ 5137us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.925016 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5006,"thread_start_us":2182,"threads_started":1}
W20260929 02:55:52.925629 22351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
I20260929 02:55:52.926117 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.922594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:39192)
0929 02:55:52.925883 (+ 3289us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.925973 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260929 02:55:52.926288 22164 leader_election.cc:334] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3079672211e419b8cf87e10fb5134a5 (127.15.18.126:43925): Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
I20260929 02:55:52.927212 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.917992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.15.18.124:44447)
0929 02:55:52.918486 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:52.918492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:52.919048 (+ 556us) server_negotiation.cc:407] Connection header received
0929 02:55:52.919908 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:52.919915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:52.919994 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:52.920065 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:52.920923 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:52.922572 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:52.924578 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:52.925917 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:52.926022 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:52.926119 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:52.926632 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 02:55:52.926778 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":216,"threads_started":1}
I20260929 02:55:52.927733 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.917761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:44447 (local address 127.0.0.1:43208)
0929 02:55:52.918934 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.918959 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:52.919716 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:52.920302 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:52.920308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:52.920345 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:52.920721 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:52.920737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:52.922782 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:52.922788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:52.924436 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:52.924448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:52.926210 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:52.926217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:52.926423 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:52.926464 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 02:55:52.927266 (+ 802us) client_negotiation.cc:240] Negotiation successful
0929 02:55:52.927397 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":843,"threads_started":1}
I20260929 02:55:52.928007 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.927460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:39194)
0929 02:55:52.927664 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.927836 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 02:55:52.928192 22351 leader_election.cc:334] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3079672211e419b8cf87e10fb5134a5 (127.15.18.126:43925): Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
I20260929 02:55:52.928404 22377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "943a13ef3a5045c582fa14727c05da9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5fff2b419dcf43688659f1bdb5541007" is_pre_election: true
I20260929 02:55:52.928679 22377 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 943a13ef3a5045c582fa14727c05da9f in term 1.
I20260929 02:55:52.929113 22163 leader_election.cc:302] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5fff2b419dcf43688659f1bdb5541007, 943a13ef3a5045c582fa14727c05da9f; no voters: f3079672211e419b8cf87e10fb5134a5
I20260929 02:55:52.929592 22909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 02:55:52.929737 22909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f3079672211e419b8cf87e10fb5134a5)
I20260929 02:55:52.929831 22909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:55:52.932520 22909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } }
I20260929 02:55:52.932971 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.923481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:33733 (local address 127.0.0.1:36890)
0929 02:55:52.925545 (+ 2064us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.925565 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:52.925678 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:52.927094 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:52.927099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:52.927138 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:52.927451 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:52.927463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:52.929124 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:52.929128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:52.931089 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:52.931103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:52.932279 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:52.932285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:52.932445 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:52.932479 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:55:52.932590 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:55:52.932723 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1975,"thread_start_us":3002,"threads_started":1}
I20260929 02:55:52.933296 22909 leader_election.cc:288] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [CANDIDATE]: Term 2 election: Requested vote from peers f3079672211e419b8cf87e10fb5134a5 (127.15.18.126:43925), 5fff2b419dcf43688659f1bdb5541007 (127.15.18.124:44447)
I20260929 02:55:52.933670 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.923037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.15.18.125:33733)
0929 02:55:52.926788 (+ 3751us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:52.926794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:52.926812 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:52.926878 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:52.926883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:52.926948 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:52.927043 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:52.927713 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:52.928995 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:52.931241 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:52.932110 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:52.932201 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:52.932294 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:52.932642 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 02:55:52.932874 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3646,"thread_start_us":3473,"threads_started":1}
I20260929 02:55:52.933753 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.933423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:39210)
0929 02:55:52.933536 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.933607 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 02:55:52.933769 22377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "943a13ef3a5045c582fa14727c05da9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5fff2b419dcf43688659f1bdb5541007"
I20260929 02:55:52.933986 22377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:55:52.934194 22189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fff2b419dcf43688659f1bdb5541007" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "943a13ef3a5045c582fa14727c05da9f" is_pre_election: true
I20260929 02:55:52.934518 22189 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5fff2b419dcf43688659f1bdb5541007 in current term 2: Already voted for candidate 943a13ef3a5045c582fa14727c05da9f in this term.
I20260929 02:55:52.935045 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.934582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:39216)
0929 02:55:52.934815 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.934909 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260929 02:55:52.935089 22353 leader_election.cc:302] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fff2b419dcf43688659f1bdb5541007; no voters: 943a13ef3a5045c582fa14727c05da9f, f3079672211e419b8cf87e10fb5134a5
W20260929 02:55:52.935191 22164 leader_election.cc:334] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f3079672211e419b8cf87e10fb5134a5 (127.15.18.126:43925): Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
I20260929 02:55:52.936517 22377 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 943a13ef3a5045c582fa14727c05da9f in term 2.
I20260929 02:55:52.936674 22910 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:55:52.936884 22163 leader_election.cc:302] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5fff2b419dcf43688659f1bdb5541007, 943a13ef3a5045c582fa14727c05da9f; no voters: f3079672211e419b8cf87e10fb5134a5
I20260929 02:55:52.937245 22909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:55:52.937611 22909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [term 2 LEADER]: Becoming Leader. State: Replica: 943a13ef3a5045c582fa14727c05da9f, State: Running, Role: LEADER
I20260929 02:55:52.937958 22909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } }
I20260929 02:55:52.939462 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 943a13ef3a5045c582fa14727c05da9f. Latest consensus state: current_term: 2 leader_uuid: "943a13ef3a5045c582fa14727c05da9f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:52.939800 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:52.940477 22921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:52.944113 22921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:52.945015 22921 catalog_manager.cc:1259] Loaded cluster ID: fb387822bc0a47e7ab62ee0e6ba510b7
I20260929 02:55:52.945091 22921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:52.946269 22921 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:52.947104 22921 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Loaded TSK: 0
I20260929 02:55:52.947997 22921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:55:52.978492 22178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:52.985837 22178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:52.990329 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.989868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:39238)
0929 02:55:52.990066 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.990175 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:55:52.990484 22377 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [term 2 FOLLOWER]: Refusing update from remote peer 943a13ef3a5045c582fa14727c05da9f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 02:55:52.991420 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:52.990952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:39246)
0929 02:55:52.991117 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:52.991203 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 02:55:52.991235 22919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 02:55:52.991617 22164 consensus_peers.cc:487] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f -> Peer f3079672211e419b8cf87e10fb5134a5 (127.15.18.126:43925): Couldn't send request to peer f3079672211e419b8cf87e10fb5134a5. Status: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:55:52.993939 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 943a13ef3a5045c582fa14727c05da9f. Latest consensus state: current_term: 2 leader_uuid: "943a13ef3a5045c582fa14727c05da9f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:52.994187 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:55:52.995544 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "943a13ef3a5045c582fa14727c05da9f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:52.995831 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:52.996094 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "943a13ef3a5045c582fa14727c05da9f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f3079672211e419b8cf87e10fb5134a5" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43925 } } peers { permanent_uuid: "943a13ef3a5045c582fa14727c05da9f" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 33733 } attrs { promote: false } } peers { permanent_uuid: "5fff2b419dcf43688659f1bdb5541007" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 44447 } attrs { promote: false } } }
I20260929 02:55:52.996364 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fff2b419dcf43688659f1bdb5541007 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:55:53.011358 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.005031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:45099 (local address 127.0.0.1:38796)
0929 02:55:53.005214 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.005236 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:53.005350 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:53.005762 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:53.005766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:53.005792 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:53.005953 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:53.005962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.007763 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.007768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:53.009424 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:53.009435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.010511 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.010521 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:53.010782 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.010819 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:55:53.010924 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 02:55:53.011054 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":34}
I20260929 02:55:53.012261 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.005082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.15.18.67:45099)
0929 02:55:53.005495 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:53.005499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:53.005514 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:55:53.005569 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:53.005572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:53.005634 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:53.005718 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:53.006508 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.007625 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.009598 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.010336 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.010442 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.010541 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:53.011856 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0929 02:55:53.012001 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":235,"threads_started":1}
I20260929 02:55:53.013504 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.005819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:45497 (local address 127.0.0.1:56054)
0929 02:55:53.006051 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.006069 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:53.006188 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:53.007108 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:53.007113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:53.007137 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:53.007496 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:53.007536 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.008972 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.008979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:53.011312 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:53.011325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.012836 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.012842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:53.012973 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.012994 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 02:55:53.013068 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:55:53.013211 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":46}
I20260929 02:55:53.013517 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.006156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.15.18.66:45497)
0929 02:55:53.006557 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:53.006563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:53.006810 (+ 247us) server_negotiation.cc:407] Connection header received
0929 02:55:53.006881 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:53.006886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:53.006958 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:53.007659 (+ 701us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:53.007826 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.008827 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.011494 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.012629 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.012739 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.012830 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:53.013106 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 02:55:53.013250 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":209,"threads_started":1}
I20260929 02:55:53.014367 22630 tablet_service.cc:1467] Processing CreateTablet for tablet 211a77afd841435abc6607469905b836 (DEFAULT_TABLE table=test-table [id=429900b288dd43689391b47ff3660e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:53.014837 22630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211a77afd841435abc6607469905b836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:53.018066 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.006415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.15.18.65:40765)
0929 02:55:53.006810 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:53.006816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:53.011684 (+ 4868us) server_negotiation.cc:407] Connection header received
0929 02:55:53.011839 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:53.011845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:53.011931 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:53.012524 (+ 593us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:53.012701 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.013872 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.015891 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.017046 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.017181 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.017282 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:53.017662 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 02:55:53.017802 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":327,"threads_started":1}
I20260929 02:55:53.018213 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.005819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:40765 (local address 127.0.0.1:58378)
0929 02:55:53.011600 (+ 5781us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.011618 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:53.011712 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:53.012111 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:53.012117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:53.012160 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:53.012380 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:53.012390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.014008 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.014013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:53.015700 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:53.015714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.017212 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.017218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:53.017494 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.017533 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:55:53.017639 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 02:55:53.017786 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5662,"mutex_wait_us":705,"threads_started":1}
I20260929 02:55:53.020979 22772 tablet_service.cc:1467] Processing CreateTablet for tablet 211a77afd841435abc6607469905b836 (DEFAULT_TABLE table=test-table [id=429900b288dd43689391b47ff3660e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:53.020977 22481 tablet_service.cc:1467] Processing CreateTablet for tablet 211a77afd841435abc6607469905b836 (DEFAULT_TABLE table=test-table [id=429900b288dd43689391b47ff3660e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:53.021420 22481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211a77afd841435abc6607469905b836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:53.021420 22772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211a77afd841435abc6607469905b836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:53.025147 22932 tablet_bootstrap.cc:492] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0: Bootstrap starting.
I20260929 02:55:53.026585 22932 tablet_bootstrap.cc:654] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:53.028558 22932 tablet_bootstrap.cc:492] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0: No bootstrap required, opened a new log
I20260929 02:55:53.028693 22932 ts_tablet_manager.cc:1397] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:55:53.029295 22932 raft_consensus.cc:348] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:53.029500 22932 raft_consensus.cc:374] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:53.029608 22932 raft_consensus.cc:729] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4258656104f48bfbebd1b16b87233d0, State: Initialized, Role: FOLLOWER
I20260929 02:55:53.029826 22932 consensus_queue.cc:260] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:53.030582 22932 ts_tablet_manager.cc:1428] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:53.030901 22935 tablet_bootstrap.cc:492] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80: Bootstrap starting.
I20260929 02:55:53.032745 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.031871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.66:37235)
0929 02:55:53.032378 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.032516 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"threads_started":1}
I20260929 02:55:53.032753 22935 tablet_bootstrap.cc:654] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:53.034621 22935 tablet_bootstrap.cc:492] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80: No bootstrap required, opened a new log
I20260929 02:55:53.034756 22935 ts_tablet_manager.cc:1397] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:55:53.035386 22935 raft_consensus.cc:348] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:53.035595 22935 raft_consensus.cc:374] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:53.035682 22935 raft_consensus.cc:729] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0628f790c74c0bad86b970bbf2bd80, State: Initialized, Role: FOLLOWER
I20260929 02:55:53.035913 22935 consensus_queue.cc:260] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:53.036120 22934 tablet_bootstrap.cc:492] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f: Bootstrap starting.
I20260929 02:55:53.036691 22935 ts_tablet_manager.cc:1428] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:53.037882 22934 tablet_bootstrap.cc:654] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:53.038017 22697 heartbeater.cc:503] Master 127.15.18.125:33733 was elected leader, sending a full tablet report...
I20260929 02:55:53.038880 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.037151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.65:58689)
0929 02:55:53.038456 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.038612 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260929 02:55:53.039124 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.038601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.66:37235)
0929 02:55:53.038914 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.038994 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260929 02:55:53.041006 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.040686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.65:56031)
0929 02:55:53.040825 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.040892 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:55:53.041771 22551 heartbeater.cc:503] Master 127.15.18.125:33733 was elected leader, sending a full tablet report...
I20260929 02:55:53.044984 22934 tablet_bootstrap.cc:492] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f: No bootstrap required, opened a new log
I20260929 02:55:53.045168 22934 ts_tablet_manager.cc:1397] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260929 02:55:53.045862 22934 raft_consensus.cc:348] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:53.046048 22934 raft_consensus.cc:374] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:53.046128 22934 raft_consensus.cc:729] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45d1de6e9294aee87b262d36df0307f, State: Initialized, Role: FOLLOWER
I20260929 02:55:53.046365 22934 consensus_queue.cc:260] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:53.047122 22934 ts_tablet_manager.cc:1428] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 02:55:53.050222 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.047498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.67:45861)
0929 02:55:53.049850 (+ 2352us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.049964 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2237,"thread_start_us":2045,"threads_started":1}
I20260929 02:55:53.051174 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.050860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.67:48557)
0929 02:55:53.051003 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.051075 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:55:53.204825 22939 raft_consensus.cc:482] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:53.205073 22939 raft_consensus.cc:504] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:53.206565 22939 leader_election.cc:288] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c45d1de6e9294aee87b262d36df0307f (127.15.18.67:45099), a4258656104f48bfbebd1b16b87233d0 (127.15.18.66:45497)
I20260929 02:55:53.213122 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.206992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:60213 (local address 127.15.18.67:45099)
0929 02:55:53.207181 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:53.207187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:53.207729 (+ 542us) server_negotiation.cc:407] Connection header received
0929 02:55:53.207883 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:53.207889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:53.207981 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:53.208071 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:53.208722 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.209657 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.211425 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.212224 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.212365 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.212440 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:53.212748 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 02:55:53.212870 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:55:53.213145 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.206983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:43763 (local address 127.15.18.66:45497)
0929 02:55:53.207197 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:53.207202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:53.207222 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:55:53.207394 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:53.207399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:53.207502 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:53.207640 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:53.208730 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.209632 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.211432 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.212223 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.212375 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.212447 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:53.212759 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 02:55:53.212891 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:55:53.213176 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.206862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:45497 (local address 127.15.18.65:43763)
0929 02:55:53.207020 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.207038 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:53.207183 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:53.207651 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:53.207654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:53.207677 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:53.207981 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:53.207993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.209766 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.209772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:53.211291 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:53.211302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.212447 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.212453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:53.212618 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.212640 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:55:53.212724 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 02:55:53.212860 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":61}
I20260929 02:55:53.213662 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.206862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:45099 (local address 127.15.18.65:60213)
0929 02:55:53.207624 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.207641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:53.207760 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:53.208176 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:53.208180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:53.208203 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:53.208577 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:53.208590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.209782 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.209786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:53.211278 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:53.211290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.212418 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.212427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:53.212590 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.212619 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:55:53.212691 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 02:55:53.212814 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":159,"thread_start_us":342,"threads_started":1}
I20260929 02:55:53.213990 22650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211a77afd841435abc6607469905b836" candidate_uuid: "0e0628f790c74c0bad86b970bbf2bd80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4258656104f48bfbebd1b16b87233d0" is_pre_election: true
I20260929 02:55:53.214351 22650 raft_consensus.cc:2454] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0628f790c74c0bad86b970bbf2bd80 in term 0.
I20260929 02:55:53.214334 22792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211a77afd841435abc6607469905b836" candidate_uuid: "0e0628f790c74c0bad86b970bbf2bd80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45d1de6e9294aee87b262d36df0307f" is_pre_election: true
I20260929 02:55:53.214537 22792 raft_consensus.cc:2454] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0628f790c74c0bad86b970bbf2bd80 in term 0.
I20260929 02:55:53.214932 22434 leader_election.cc:302] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e0628f790c74c0bad86b970bbf2bd80, a4258656104f48bfbebd1b16b87233d0, c45d1de6e9294aee87b262d36df0307f; no voters:
I20260929 02:55:53.215337 22939 raft_consensus.cc:2781] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:53.215444 22939 raft_consensus.cc:482] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:53.215557 22939 raft_consensus.cc:3037] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:53.217702 22939 raft_consensus.cc:504] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:53.218335 22939 leader_election.cc:288] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [CANDIDATE]: Term 1 election: Requested vote from peers c45d1de6e9294aee87b262d36df0307f (127.15.18.67:45099), a4258656104f48bfbebd1b16b87233d0 (127.15.18.66:45497)
I20260929 02:55:53.218861 22650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211a77afd841435abc6607469905b836" candidate_uuid: "0e0628f790c74c0bad86b970bbf2bd80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4258656104f48bfbebd1b16b87233d0"
I20260929 02:55:53.218876 22792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211a77afd841435abc6607469905b836" candidate_uuid: "0e0628f790c74c0bad86b970bbf2bd80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c45d1de6e9294aee87b262d36df0307f"
I20260929 02:55:53.219040 22650 raft_consensus.cc:3037] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:53.219064 22792 raft_consensus.cc:3037] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:53.221161 22792 raft_consensus.cc:2454] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0628f790c74c0bad86b970bbf2bd80 in term 1.
I20260929 02:55:53.221161 22650 raft_consensus.cc:2454] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0628f790c74c0bad86b970bbf2bd80 in term 1.
I20260929 02:55:53.221596 22434 leader_election.cc:302] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0628f790c74c0bad86b970bbf2bd80, a4258656104f48bfbebd1b16b87233d0; no voters:
I20260929 02:55:53.221937 22939 raft_consensus.cc:2781] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:53.222365 22939 raft_consensus.cc:686] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 1 LEADER]: Becoming Leader. State: Replica: 0e0628f790c74c0bad86b970bbf2bd80, State: Running, Role: LEADER
I20260929 02:55:53.222751 22939 consensus_queue.cc:237] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } } peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } }
I20260929 02:55:53.223374 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.222959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.65:52153)
0929 02:55:53.223131 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.223236 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 02:55:53.226465 22178 catalog_manager.cc:5495] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e0628f790c74c0bad86b970bbf2bd80 (127.15.18.65). New cstate: current_term: 1 leader_uuid: "0e0628f790c74c0bad86b970bbf2bd80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e0628f790c74c0bad86b970bbf2bd80" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40765 } health_report { overall_health: HEALTHY } } }
I20260929 02:55:53.257624 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.252392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.15.18.65:40765)
0929 02:55:53.252723 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:53.252727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:53.252746 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:55:53.252834 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:53.252838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:53.252918 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:53.253054 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:53.253383 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.254164 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.255613 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.256062 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.256176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.256552 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:53.257100 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:53.257219 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:53.257358 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 02:55:53.257423 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260929 02:55:53.261492 22792 raft_consensus.cc:1261] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f [term 1 FOLLOWER]: Refusing update from remote peer 0e0628f790c74c0bad86b970bbf2bd80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:53.261830 22650 raft_consensus.cc:1261] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0 [term 1 FOLLOWER]: Refusing update from remote peer 0e0628f790c74c0bad86b970bbf2bd80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:53.262239 22945 consensus_queue.cc:1035] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45d1de6e9294aee87b262d36df0307f" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 45099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:53.262600 22939 consensus_queue.cc:1035] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4258656104f48bfbebd1b16b87233d0" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 45497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:53.263993 22839 heartbeater.cc:503] Master 127.15.18.125:33733 was elected leader, sending a full tablet report...
I20260929 02:55:53.265575 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.264782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.67:42727)
0929 02:55:53.265227 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.265350 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333}
I20260929 02:55:53.266716 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.266257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.66:52183)
0929 02:55:53.266479 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.266551 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260929 02:55:53.274072 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.273425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.65:38195)
0929 02:55:53.273762 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.273860 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260929 02:55:53.274398 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.273921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.67:53631)
0929 02:55:53.274140 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.274230 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 02:55:53.278275 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.277830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.15.18.66:54627)
0929 02:55:53.278040 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.278124 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 02:55:53.280095 22950 mvcc.cc:205] Tried to move back new op lower bound from 7334504666152505344 to 7334504666004606976. Current Snapshot: MvccSnapshot[applied={T|T < 7334504666152505344}]
I20260929 02:55:53.288143 22179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:53.288437 22179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:53.294364 22179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 943a13ef3a5045c582fa14727c05da9f: Sending DeleteTablet for 3 replicas of tablet 211a77afd841435abc6607469905b836
I20260929 02:55:53.295553 22772 tablet_service.cc:1514] Processing DeleteTablet for tablet 211a77afd841435abc6607469905b836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:53 UTC) from {username='oryx', principal='oryx/127.15.18.125@KRBTEST.COM'} at 127.0.0.1:38796
I20260929 02:55:53.295816 22481 tablet_service.cc:1514] Processing DeleteTablet for tablet 211a77afd841435abc6607469905b836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:53 UTC) from {username='oryx', principal='oryx/127.15.18.125@KRBTEST.COM'} at 127.0.0.1:58378
I20260929 02:55:53.295989 22630 tablet_service.cc:1514] Processing DeleteTablet for tablet 211a77afd841435abc6607469905b836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:53 UTC) from {username='oryx', principal='oryx/127.15.18.125@KRBTEST.COM'} at 127.0.0.1:56054
I20260929 02:55:53.297178 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 22356
I20260929 02:55:53.312252 22960 tablet_replica.cc:330] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0: stopping tablet replica
I20260929 02:55:53.312500 22960 raft_consensus.cc:2229] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:53.312879 22960 raft_consensus.cc:2258] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:53.315275 22958 tablet_replica.cc:330] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f: stopping tablet replica
I20260929 02:55:53.315660 22958 raft_consensus.cc:2229] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:53.316058 22958 raft_consensus.cc:2258] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:53.317381 22960 ts_tablet_manager.cc:1905] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:53.317553 22958 ts_tablet_manager.cc:1905] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:53.323868 22959 tablet_replica.cc:330] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80: stopping tablet replica
I20260929 02:55:53.324113 22960 ts_tablet_manager.cc:1918] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:53.324102 22959 raft_consensus.cc:2229] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:53.324268 22960 log.cc:1196] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/211a77afd841435abc6607469905b836
I20260929 02:55:53.324658 22958 ts_tablet_manager.cc:1918] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:53.324775 22958 log.cc:1196] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/211a77afd841435abc6607469905b836
I20260929 02:55:53.324774 22960 ts_tablet_manager.cc:1939] T 211a77afd841435abc6607469905b836 P a4258656104f48bfbebd1b16b87233d0: Deleting consensus metadata
I20260929 02:55:53.324783 22959 raft_consensus.cc:2258] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:53.325181 22958 ts_tablet_manager.cc:1939] T 211a77afd841435abc6607469905b836 P c45d1de6e9294aee87b262d36df0307f: Deleting consensus metadata
I20260929 02:55:53.326153 22163 catalog_manager.cc:4841] TS a4258656104f48bfbebd1b16b87233d0 (127.15.18.66:45497): tablet 211a77afd841435abc6607469905b836 (table test-table [id=429900b288dd43689391b47ff3660e50]) successfully deleted
I20260929 02:55:53.326273 22959 ts_tablet_manager.cc:1905] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:53.326531 22165 catalog_manager.cc:4841] TS c45d1de6e9294aee87b262d36df0307f (127.15.18.67:45099): tablet 211a77afd841435abc6607469905b836 (table test-table [id=429900b288dd43689391b47ff3660e50]) successfully deleted
I20260929 02:55:53.332984 22959 ts_tablet_manager.cc:1918] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:53.333153 22959 log.cc:1196] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/211a77afd841435abc6607469905b836
I20260929 02:55:53.333371 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.300022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.15.18.65:40765)
0929 02:55:53.300327 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:53.300334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:53.308975 (+ 8641us) server_negotiation.cc:407] Connection header received
0929 02:55:53.309516 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:53.309524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:53.309645 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:53.309732 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:53.313190 (+ 3458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.314421 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.316566 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.318069 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.318236 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.325215 (+ 6979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:53.325234 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:53.325254 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:53.325314 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:53.329410 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:53.330411 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:53.330426 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:53.330429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:53.330642 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:53.331137 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:53.331148 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:53.331152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:53.331721 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:53.332087 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:53.332714 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 02:55:53.332938 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260929 02:55:53.333634 22959 ts_tablet_manager.cc:1939] T 211a77afd841435abc6607469905b836 P 0e0628f790c74c0bad86b970bbf2bd80: Deleting consensus metadata
I20260929 02:55:53.334020 22541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:53.438687 22165 catalog_manager.cc:4841] TS 0e0628f790c74c0bad86b970bbf2bd80 (127.15.18.65:40765): tablet 211a77afd841435abc6607469905b836 (table test-table [id=429900b288dd43689391b47ff3660e50]) successfully deleted
I20260929 02:55:53.438987 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 22356
I20260929 02:55:53.456554 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 22558
I20260929 02:55:53.469502 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.457313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.15.18.66:45497)
0929 02:55:53.457478 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:53.457484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:53.457547 (+ 63us) server_negotiation.cc:407] Connection header received
0929 02:55:53.457733 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:53.457738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:53.457800 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:53.457910 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:53.459135 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.459948 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.461144 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.461525 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.461635 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.464116 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:53.464131 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:53.464147 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:53.464183 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:53.466857 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:53.467489 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:53.467497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:53.467499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:53.467651 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:53.467988 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:53.467992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:53.467996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:53.468394 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:53.468683 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:53.468974 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 02:55:53.469156 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:55:53.470484 22690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:53.492918 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.492481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:39252)
0929 02:55:53.492664 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.492771 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 02:55:53.494081 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.493749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43925 (local address 127.0.0.1:39266)
0929 02:55:53.493895 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:53.493967 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 02:55:53.588617 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 22558
I20260929 02:55:53.606333 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 22708
I20260929 02:55:53.619356 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:53.607158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.15.18.67:45099)
0929 02:55:53.607343 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:53.607347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:53.607368 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:55:53.607603 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:53.607607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:53.607686 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:53.607773 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:53.609133 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.609901 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.611072 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:53.611465 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:53.611610 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:53.614053 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:53.614066 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:53.614082 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:53.614126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:53.616865 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:53.617496 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:53.617503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:53.617505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:53.617635 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:53.618049 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:53.618052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:53.618054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:53.618411 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:53.618618 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:53.618914 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 02:55:53.619078 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:55:53.620044 22832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:53.712106 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 22708
I20260929 02:55:53.726317 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 22042
I20260929 02:55:53.727324 22209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:53.849298 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 22042
I20260929 02:55:53.866255 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 22143
I20260929 02:55:53.867271 22397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:53.980731 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 22143
2026-09-29T02:55:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9337 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:55:54 dist-test-slave-q5f0 krb5kdc[22970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:55:54 dist-test-slave-q5f0 krb5kdc[22970](info): set up 2 sockets
Sep 29 02:55:54 dist-test-slave-q5f0 krb5kdc[22970](info): commencing operation
krb5kdc: starting...
W20260929 02:55:54.984436 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:55:55 dist-test-slave-q5f0 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:55:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.126.keytab.
Entry for principal oryx/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.126.keytab.
I20260929 02:55:55.199893 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:44301
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.126.keytab
--principal=oryx/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:44301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:55.517647 22983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:55.517990 22983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:55.518121 22983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:55.525481 22983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:55:55.525622 22983 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:55:55.525698 22983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:55.525780 22983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:55.525841 22983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:55:55.525923 22983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:55:55.536428 22983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:44301
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.126.keytab
--principal=oryx/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:44301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:55.537750 22983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:55.539093 22983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:55.550850 22988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:55.550884 22989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:55.550909 22991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:55.551332 22983 server_base.cc:1034] running on GCE node
I20260929 02:55:55.552351 22983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:55.553776 22983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:55.555007 22983 hybrid_clock.cc:648] HybridClock initialized: now 1790650555554951 us; error 46 us; skew 500 ppm
I20260929 02:55:55.555367 22983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:55 dist-test-slave-q5f0 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650555, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:55.558403 22983 init.cc:377] Logged in from keytab as oryx/127.15.18.126@KRBTEST.COM (short username oryx)
I20260929 02:55:55.559906 22983 webserver.cc:460] Webserver started at http://127.15.18.126:41399/ using document root <none> and password file <none>
I20260929 02:55:55.560570 22983 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:55.560653 22983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:55.560926 22983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:55.563848 22983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "bf57cd62707e49dbbb54e123056e9cfb"
format_stamp: "Formatted at 2026-09-29 02:55:55 on dist-test-slave-q5f0"
server_key: "42f377a6917bb34da9b3328112714ebb"
server_key_iv: "ddc2b3e43be89c2896e24a0b27e05dfd"
server_key_version: "encryptionkey@0"
I20260929 02:55:55.564635 22983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "bf57cd62707e49dbbb54e123056e9cfb"
format_stamp: "Formatted at 2026-09-29 02:55:55 on dist-test-slave-q5f0"
server_key: "42f377a6917bb34da9b3328112714ebb"
server_key_iv: "ddc2b3e43be89c2896e24a0b27e05dfd"
server_key_version: "encryptionkey@0"
I20260929 02:55:55.569626 22983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:55:55.573101 22998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:55.574548 22983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:55:55.574698 22983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "bf57cd62707e49dbbb54e123056e9cfb"
format_stamp: "Formatted at 2026-09-29 02:55:55 on dist-test-slave-q5f0"
server_key: "42f377a6917bb34da9b3328112714ebb"
server_key_iv: "ddc2b3e43be89c2896e24a0b27e05dfd"
server_key_version: "encryptionkey@0"
I20260929 02:55:55.574887 22983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:55.601060 22983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:55.605708 22983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:55.605947 22983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:55.625996 22983 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:44301
I20260929 02:55:55.626009 23059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:44301 every 8 connection(s)
I20260929 02:55:55.627323 22983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:55:55.630334 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 22983
I20260929 02:55:55.630546 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:55:55.631321 15433 external_mini_cluster.cc:1428] Setting key 68d95d8cbb519967839918ab385b6491
I20260929 02:55:55.631654 23060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:55.646390 23060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb: Bootstrap starting.
Sep 29 02:55:55 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:55.649760 23060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:55.650866 23060 log.cc:824] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:55.653695 23060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb: No bootstrap required, opened a new log
I20260929 02:55:55.655184 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:55.633329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.15.18.126:44301)
0929 02:55:55.634112 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:55.634128 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:55.634172 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:55:55.636352 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:55.636377 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:55.636811 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:55.637167 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:55.639035 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:55.640453 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:55.642287 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:55.643116 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:55.643276 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:55.647905 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:55.647917 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:55.647934 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:55.647988 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:55.650980 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:55.651774 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:55.651782 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:55.651789 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:55.651979 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:55.652350 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:55.652355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:55.652359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:55.652885 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:55.653168 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:55.653608 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 02:55:55.654178 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":390,"threads_started":1}
I20260929 02:55:55.661128 23060 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf57cd62707e49dbbb54e123056e9cfb" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44301 } }
I20260929 02:55:55.661486 23060 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:55.661577 23060 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf57cd62707e49dbbb54e123056e9cfb, State: Initialized, Role: FOLLOWER
I20260929 02:55:55.662082 23060 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf57cd62707e49dbbb54e123056e9cfb" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44301 } }
I20260929 02:55:55.662308 23060 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:55:55.662423 23060 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:55:55.662570 23060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:55.664789 23060 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf57cd62707e49dbbb54e123056e9cfb" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44301 } }
I20260929 02:55:55.665282 23060 leader_election.cc:302] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf57cd62707e49dbbb54e123056e9cfb; no voters:
I20260929 02:55:55.665854 23060 leader_election.cc:288] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:55:55.666030 23065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:55.667057 23065 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [term 1 LEADER]: Becoming Leader. State: Replica: bf57cd62707e49dbbb54e123056e9cfb, State: Running, Role: LEADER
I20260929 02:55:55.667603 23065 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf57cd62707e49dbbb54e123056e9cfb" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44301 } }
I20260929 02:55:55.668011 23060 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:55:55.670346 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf57cd62707e49dbbb54e123056e9cfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf57cd62707e49dbbb54e123056e9cfb" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44301 } } }
I20260929 02:55:55.670368 23067 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf57cd62707e49dbbb54e123056e9cfb. Latest consensus state: current_term: 1 leader_uuid: "bf57cd62707e49dbbb54e123056e9cfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf57cd62707e49dbbb54e123056e9cfb" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44301 } } }
I20260929 02:55:55.670930 23067 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:55.670753 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb [sys.catalog]: This master's current role is: LEADER
I20260929 02:55:55.673578 23074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:55:55.678885 23074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:55:55.687695 23074 catalog_manager.cc:1347] Generated new cluster ID: 57e77e242ad5426ead2bf1c734255c89
I20260929 02:55:55.687819 23074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:55:55.695204 23074 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:55:55.697086 23074 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:55:55.704298 23074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb: Generated new TSK 0
I20260929 02:55:55.705318 23074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab.
Entry for principal oryx/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab.
I20260929 02:55:55.752064 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:44301
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab
--principal=oryx/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:56.073660 23086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:56.074002 23086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:56.074124 23086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:56.081801 23086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:56.081916 23086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:56.082000 23086 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:55:56.082077 23086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:55:56.091897 23086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab
--principal=oryx/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.18.126:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:56.093101 23086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:56.094406 23086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:56.107401 23092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:56.107482 23094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:56.107545 23091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:56.108067 23086 server_base.cc:1034] running on GCE node
I20260929 02:55:56.108732 23086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:56.109758 23086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:56.110983 23086 hybrid_clock.cc:648] HybridClock initialized: now 1790650556110927 us; error 32 us; skew 500 ppm
I20260929 02:55:56.111265 23086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:56 dist-test-slave-q5f0 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650556, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:56.114215 23086 init.cc:377] Logged in from keytab as oryx/127.15.18.65@KRBTEST.COM (short username oryx)
I20260929 02:55:56.115617 23086 webserver.cc:460] Webserver started at http://127.15.18.65:43839/ using document root <none> and password file <none>
I20260929 02:55:56.116302 23086 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:56.116400 23086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:56.116700 23086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:56.119503 23086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "fdef5fa80ed941cc8fd4932698bef4e5"
format_stamp: "Formatted at 2026-09-29 02:55:56 on dist-test-slave-q5f0"
server_key: "8db52dd1a4a56721297ba555a5c32292"
server_key_iv: "25d68fd1f06e0d976373a18e905e10eb"
server_key_version: "encryptionkey@0"
I20260929 02:55:56.120312 23086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "fdef5fa80ed941cc8fd4932698bef4e5"
format_stamp: "Formatted at 2026-09-29 02:55:56 on dist-test-slave-q5f0"
server_key: "8db52dd1a4a56721297ba555a5c32292"
server_key_iv: "25d68fd1f06e0d976373a18e905e10eb"
server_key_version: "encryptionkey@0"
I20260929 02:55:56.125397 23086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:55:56.129092 23101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:56.130581 23086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:55:56.130746 23086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "fdef5fa80ed941cc8fd4932698bef4e5"
format_stamp: "Formatted at 2026-09-29 02:55:56 on dist-test-slave-q5f0"
server_key: "8db52dd1a4a56721297ba555a5c32292"
server_key_iv: "25d68fd1f06e0d976373a18e905e10eb"
server_key_version: "encryptionkey@0"
I20260929 02:55:56.130987 23086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:56.148185 23086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:56.152321 23086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:56.152572 23086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:56.155194 23086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:56.155282 23086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:56.155417 23086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:56.155479 23086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:55:56 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650556, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:56.178928 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:56.156017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:47549 (local address 127.15.18.126:44301)
0929 02:55:56.156215 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:56.156222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:56.157665 (+ 1443us) server_negotiation.cc:407] Connection header received
0929 02:55:56.160014 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:56.160022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:56.160124 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:56.160237 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:56.162464 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.163434 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.166437 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.167043 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.167189 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:56.172030 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:56.172038 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:56.172043 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:56.172090 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:56.175174 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:56.176227 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:56.176233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:56.176237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:56.176404 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:56.176991 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:56.176995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:56.176997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:56.177293 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:56.177403 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:56.178472 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0929 02:55:56.178682 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:55:56.180855 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:56.156588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44301 (local address 127.15.18.65:47549)
0929 02:55:56.157467 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:56.157524 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:56.159619 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:56.160550 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:56.160564 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:56.161364 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:56.162269 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:56.162296 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.163635 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.163644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:56.166238 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:56.166261 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.167328 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.167345 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:56.167734 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:56.168683 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:56.168716 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:56.171704 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:56.175364 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:56.175376 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:56.175399 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:56.176099 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:56.176538 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:56.176548 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:56.176552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:56.176867 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:56.177405 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:56.177415 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:56.177945 (+ 530us) client_negotiation.cc:769] Sending connection context
0929 02:55:56.178329 (+ 384us) client_negotiation.cc:240] Negotiation successful
0929 02:55:56.178955 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":355,"threads_started":1}
I20260929 02:55:56.200786 23086 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:36383
I20260929 02:55:56.200798 23220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:36383 every 8 connection(s)
I20260929 02:55:56.202423 23086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:55:56.204581 23221 heartbeater.cc:347] Connected to a master server at 127.15.18.126:44301
I20260929 02:55:56.204810 23221 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:56.205555 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:56.191127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44301 (local address 127.15.18.65:46225)
0929 02:55:56.191773 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:56.191794 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:56.191934 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:56.192344 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:56.192348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:56.192807 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:56.193089 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:56.193098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.194616 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.194620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:56.195803 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:56.195817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.196742 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.196750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:56.196950 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:56.197823 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:56.197855 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:56.199045 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:56.202643 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:56.202648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:56.202651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:56.203082 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:56.203487 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:56.203492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:56.203495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:56.203739 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:56.204649 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:56.204652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:56.204830 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 02:55:56.204953 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 02:55:56.205203 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":329,"threads_started":1}
I20260929 02:55:56.205627 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:56.191177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:46225 (local address 127.15.18.126:44301)
0929 02:55:56.191377 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:56.191384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:56.191889 (+ 505us) server_negotiation.cc:407] Connection header received
0929 02:55:56.192047 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:56.192053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:56.192153 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:56.192281 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:56.193224 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.194446 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.195969 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.196518 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.196661 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:56.199261 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:56.199266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:56.199269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:56.199301 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:56.202499 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:56.203210 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:56.203216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:56.203219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:56.203360 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:56.204132 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:56.204137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:56.204141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:56.204437 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:56.204597 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:56.205087 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 02:55:56.205308 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:55:56.205652 23221 heartbeater.cc:511] Master 127.15.18.126:44301 requested a full tablet report, sending...
I20260929 02:55:56.207944 23015 ts_manager.cc:194] Registered new tserver with Master: fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383)
I20260929 02:55:56.209515 23015 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:47549
I20260929 02:55:56.212224 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 23086
I20260929 02:55:56.212386 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:55:56.212953 15433 external_mini_cluster.cc:1428] Setting key a79f07fb8e8f4d0b03518f7f8fe908b8
WARNING: no policy specified for oryx/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab.
Entry for principal oryx/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab.
I20260929 02:55:56.258497 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:44301
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab
--principal=oryx/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:56.576434 23225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:56.576741 23225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:56.576853 23225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:56.584230 23225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:56.584347 23225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:56.584429 23225 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:55:56.584506 23225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:55:56.594386 23225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab
--principal=oryx/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.18.126:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:56.595681 23225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:56.596999 23225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:56.609946 23230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:56.610005 23233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:56.609946 23231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:56.610790 23225 server_base.cc:1034] running on GCE node
I20260929 02:55:56.611464 23225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:56.612493 23225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:56.613758 23225 hybrid_clock.cc:648] HybridClock initialized: now 1790650556613687 us; error 50 us; skew 500 ppm
I20260929 02:55:56.614060 23225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:56 dist-test-slave-q5f0 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650556, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:56.620545 23225 init.cc:377] Logged in from keytab as oryx/127.15.18.66@KRBTEST.COM (short username oryx)
I20260929 02:55:56.622203 23225 webserver.cc:460] Webserver started at http://127.15.18.66:38639/ using document root <none> and password file <none>
I20260929 02:55:56.622907 23225 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:56.623008 23225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:56.623335 23225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:56.626402 23225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "ec2cf3e1cb5841708d8a88efb52eacd3"
format_stamp: "Formatted at 2026-09-29 02:55:56 on dist-test-slave-q5f0"
server_key: "023d15d98f55614a69a74fdff488859b"
server_key_iv: "ff4cbaa1c057b40dbc394b4edd30c6b7"
server_key_version: "encryptionkey@0"
I20260929 02:55:56.627151 23225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "ec2cf3e1cb5841708d8a88efb52eacd3"
format_stamp: "Formatted at 2026-09-29 02:55:56 on dist-test-slave-q5f0"
server_key: "023d15d98f55614a69a74fdff488859b"
server_key_iv: "ff4cbaa1c057b40dbc394b4edd30c6b7"
server_key_version: "encryptionkey@0"
I20260929 02:55:56.632440 23225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:55:56.636088 23240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:56.637753 23225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:55:56.637931 23225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "ec2cf3e1cb5841708d8a88efb52eacd3"
format_stamp: "Formatted at 2026-09-29 02:55:56 on dist-test-slave-q5f0"
server_key: "023d15d98f55614a69a74fdff488859b"
server_key_iv: "ff4cbaa1c057b40dbc394b4edd30c6b7"
server_key_version: "encryptionkey@0"
I20260929 02:55:56.638170 23225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:56.654031 23225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:56.657991 23225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:56.658209 23225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:56.660727 23225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:56.660826 23225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:56.660960 23225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:56.661018 23225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:55:56 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650556, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:56.686594 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:56.661964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44301 (local address 127.15.18.66:57835)
0929 02:55:56.662824 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:56.662877 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:56.665049 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:56.665916 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:56.665927 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:56.666729 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:56.667663 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:56.667684 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.669080 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.669085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:56.671270 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:56.671289 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.672485 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.672490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:56.672791 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:56.673749 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:56.673774 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:56.677165 (+ 3391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:56.681110 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:56.681122 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:56.681154 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:56.681762 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:56.682258 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:56.682263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:56.682272 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:56.682579 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:56.683370 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:56.683380 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:56.683887 (+ 507us) client_negotiation.cc:769] Sending connection context
0929 02:55:56.684281 (+ 394us) client_negotiation.cc:240] Negotiation successful
0929 02:55:56.684899 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":378,"threads_started":1}
I20260929 02:55:56.687778 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:56.661443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:57835 (local address 127.15.18.126:44301)
0929 02:55:56.661626 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:56.661632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:56.663023 (+ 1391us) server_negotiation.cc:407] Connection header received
0929 02:55:56.665401 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:56.665408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:56.665503 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:56.665612 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:56.667837 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.668932 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.671417 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.672050 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.672180 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:56.677495 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:56.677503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:56.677509 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:56.677559 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:56.680943 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:56.681909 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:56.681915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:56.681919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:56.682099 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:56.682749 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:56.682755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:56.682759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:56.683206 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:56.683387 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:56.687211 (+ 3824us) server_negotiation.cc:299] Negotiation successful
0929 02:55:56.687444 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:55:56.699671 23225 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:36045
I20260929 02:55:56.699685 23359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:36045 every 8 connection(s)
I20260929 02:55:56.700975 23225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:55:56.703200 23360 heartbeater.cc:347] Connected to a master server at 127.15.18.126:44301
I20260929 02:55:56.703411 23360 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:56.704048 23360 heartbeater.cc:511] Master 127.15.18.126:44301 requested a full tablet report, sending...
I20260929 02:55:56.705348 23015 ts_manager.cc:194] Registered new tserver with Master: ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045)
I20260929 02:55:56.706279 23015 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:57835
I20260929 02:55:56.708741 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:56.695061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:58481 (local address 127.15.18.126:44301)
0929 02:55:56.695274 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:56.695281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:56.696177 (+ 896us) server_negotiation.cc:407] Connection header received
0929 02:55:56.696427 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:56.696432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:56.696502 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:56.696621 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:56.697953 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.698862 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.700296 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.700714 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.700820 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:56.703228 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:56.703233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:56.703236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:56.703267 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:56.706092 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:56.706833 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:56.706839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:56.706844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:56.706976 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:56.707304 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:56.707307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:56.707309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:56.707662 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:56.707804 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:56.708189 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 02:55:56.708398 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:55:56.708936 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:56.695181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44301 (local address 127.15.18.66:58481)
0929 02:55:56.696045 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:56.696070 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:56.696241 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:56.696683 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:56.696688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:56.697372 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:56.697803 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:56.697821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.698992 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.698998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:56.700154 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:56.700165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:56.700888 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:56.700895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:56.701052 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:56.701734 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:56.701754 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:56.703065 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:56.706217 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:56.706223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:56.706227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:56.706697 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:56.707080 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:56.707085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:56.707089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:56.707217 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:56.707794 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:56.707799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:56.707965 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 02:55:56.708091 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 02:55:56.708398 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":523,"threads_started":1}
I20260929 02:55:56.708937 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 23225
I20260929 02:55:56.709095 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:55:56.709609 15433 external_mini_cluster.cc:1428] Setting key 28173ff3a57f4b60438d65f5dea2afb1
WARNING: no policy specified for oryx/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab.
Entry for principal oryx/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab.
I20260929 02:55:56.752828 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:44301
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab
--principal=oryx/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:55:57.068291 23364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:55:57.068662 23364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:55:57.068830 23364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:55:57.076367 23364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:55:57.076486 23364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:55:57.076598 23364 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:55:57.076753 23364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:55:57.086870 23364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab
--principal=oryx/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.18.126:44301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:55:57.088193 23364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:55:57.089542 23364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:55:57.102384 23369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:57.102404 23370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:55:57.102499 23372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:55:57.103273 23364 server_base.cc:1034] running on GCE node
I20260929 02:55:57.104038 23364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:55:57.104936 23364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:55:57.106187 23364 hybrid_clock.cc:648] HybridClock initialized: now 1790650557106100 us; error 48 us; skew 500 ppm
I20260929 02:55:57.106515 23364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:55:57 dist-test-slave-q5f0 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650557, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:55:57.109894 23364 init.cc:377] Logged in from keytab as oryx/127.15.18.67@KRBTEST.COM (short username oryx)
I20260929 02:55:57.111377 23364 webserver.cc:460] Webserver started at http://127.15.18.67:37837/ using document root <none> and password file <none>
I20260929 02:55:57.112112 23364 fs_manager.cc:359] Metadata directory not provided
I20260929 02:55:57.112200 23364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:55:57.112476 23364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:55:57.115321 23364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "efb1b689a4b54036a4ee37dfe35c1f7a"
format_stamp: "Formatted at 2026-09-29 02:55:57 on dist-test-slave-q5f0"
server_key: "f96feaa99600198255fc97e1f57a03d6"
server_key_iv: "a97ff3d3d0a77014d2d33f2096fd0a94"
server_key_version: "encryptionkey@0"
I20260929 02:55:57.116122 23364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "efb1b689a4b54036a4ee37dfe35c1f7a"
format_stamp: "Formatted at 2026-09-29 02:55:57 on dist-test-slave-q5f0"
server_key: "f96feaa99600198255fc97e1f57a03d6"
server_key_iv: "a97ff3d3d0a77014d2d33f2096fd0a94"
server_key_version: "encryptionkey@0"
I20260929 02:55:57.121240 23364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:55:57.124737 23379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:57.126253 23364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:55:57.126420 23364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "efb1b689a4b54036a4ee37dfe35c1f7a"
format_stamp: "Formatted at 2026-09-29 02:55:57 on dist-test-slave-q5f0"
server_key: "f96feaa99600198255fc97e1f57a03d6"
server_key_iv: "a97ff3d3d0a77014d2d33f2096fd0a94"
server_key_version: "encryptionkey@0"
I20260929 02:55:57.126657 23364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:55:57.147543 23364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:55:57.152423 23364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:55:57.152726 23364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:55:57.155702 23364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:55:57.155812 23364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:55:57.155979 23364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:55:57.156119 23364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 02:55:57 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650557, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:57.177881 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:57.156183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:58023 (local address 127.15.18.126:44301)
0929 02:55:57.156402 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:57.156409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:57.157811 (+ 1402us) server_negotiation.cc:407] Connection header received
0929 02:55:57.159874 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:57.159879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:57.159980 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:57.160068 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:57.162347 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:57.163329 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:57.165759 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:57.166303 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:57.166420 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:57.171091 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:57.171098 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:57.171101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:57.171135 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:57.174258 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:57.175312 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:57.175318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:57.175322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:57.175468 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:57.176080 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:57.176086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:57.176090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:57.176370 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:57.176505 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:57.177276 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 02:55:57.177521 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:55:57.178956 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:57.156756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44301 (local address 127.15.18.67:58023)
0929 02:55:57.157594 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:57.157651 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:57.159580 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:57.160323 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:57.160332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:57.161071 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:57.162125 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:57.162157 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:57.163486 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:57.163496 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:57.165596 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:57.165609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:57.166536 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:57.166545 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:57.166872 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:57.167625 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:57.167686 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:57.170844 (+ 3158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:57.174430 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:57.174440 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:57.174464 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:57.175122 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:57.175596 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:57.175602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:57.175606 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:57.175953 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:57.176503 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:57.176512 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:57.176904 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 02:55:57.177144 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 02:55:57.177698 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":341,"threads_started":1}
I20260929 02:55:57.200693 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:57.184545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:55605 (local address 127.15.18.126:44301)
0929 02:55:57.184753 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:57.184760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:57.185702 (+ 942us) server_negotiation.cc:407] Connection header received
0929 02:55:57.185933 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:57.185940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:57.186029 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:57.186135 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:57.187192 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:57.188321 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:57.190582 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:57.191073 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:57.191210 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:57.193977 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:57.193983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:57.193986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:57.194019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:57.197135 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:57.198213 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:57.198220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:57.198223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:57.198431 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:57.198924 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:57.198929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:57.198932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:57.199282 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:57.199461 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:57.200106 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 02:55:57.200339 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:55:57.201629 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:57.184795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44301 (local address 127.15.18.67:55605)
0929 02:55:57.185558 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:57.185585 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:57.185784 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:57.186201 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:57.186207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:57.186678 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:57.187003 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:57.187013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:57.188541 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:57.188547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:57.190418 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:57.190438 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:57.191300 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:57.191308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:57.191549 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:57.192422 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:57.192455 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:57.193767 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:57.197329 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:57.197336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:57.197341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:57.198070 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:57.198575 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:57.198582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:57.198587 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:57.198792 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:57.199435 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:57.199441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:57.199804 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 02:55:57.200002 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 02:55:57.200550 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":412,"threads_started":1}
I20260929 02:55:57.207780 23364 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:34653
I20260929 02:55:57.207813 23498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:34653 every 8 connection(s)
I20260929 02:55:57.208916 23364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:55:57.210968 23499 heartbeater.cc:347] Connected to a master server at 127.15.18.126:44301
I20260929 02:55:57.211164 23499 heartbeater.cc:464] Registering TS with master...
I20260929 02:55:57.211833 23499 heartbeater.cc:511] Master 127.15.18.126:44301 requested a full tablet report, sending...
I20260929 02:55:57.212162 23221 heartbeater.cc:503] Master 127.15.18.126:44301 was elected leader, sending a full tablet report...
I20260929 02:55:57.213130 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 23364
I20260929 02:55:57.213315 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:55:57.213430 23014 ts_manager.cc:194] Registered new tserver with Master: efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67:34653)
I20260929 02:55:57.213894 15433 external_mini_cluster.cc:1428] Setting key d345c083bc2a33a87fd6bdcbdf5029fc
I20260929 02:55:57.214488 23014 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:58023
I20260929 02:55:57.217097 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:55:57 dist-test-slave-q5f0 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:55:57 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:55:57 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:55:57.709491 23360 heartbeater.cc:503] Master 127.15.18.126:44301 was elected leader, sending a full tablet report...
I20260929 02:55:58.217347 23499 heartbeater.cc:503] Master 127.15.18.126:44301 was elected leader, sending a full tablet report...
W20260929 02:55:58.233536 23507 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.972s user 0.005s sys 0.008s
W20260929 02:55:58.234740 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:55:57.255888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.15.18.126:44301)
0929 02:55:57.256112 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:57.256118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:57.256299 (+ 181us) server_negotiation.cc:407] Connection header received
0929 02:55:57.256481 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:57.256486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:57.256547 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:57.256629 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:57.257889 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:57.258605 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:57.259890 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:57.260315 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:57.260422 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.234455 (+974033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35498: BlockingRecv error: recv got EOF from 127.0.0.1:35498 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 02:55:58 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:58.252331 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.238169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.15.18.126:44301)
0929 02:55:58.238385 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.238391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.238631 (+ 240us) server_negotiation.cc:407] Connection header received
0929 02:55:58.238801 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.238805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.238891 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.238987 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:58.240385 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.241151 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.242586 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.243086 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.243181 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.246841 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:58.246846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:58.246849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:58.246885 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:58.249449 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:58.250227 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:58.250232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:58.250235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:58.250380 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:58.250822 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:58.250827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:58.250829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:58.251207 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:58.251420 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.251803 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.251985 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:55:58.256414 23014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:58.263019 23014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:58.297467 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.275484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.15.18.65:36383)
0929 02:55:58.287452 (+ 11968us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.287463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.288966 (+ 1503us) server_negotiation.cc:407] Connection header received
0929 02:55:58.289078 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.289085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.289422 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.289618 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:58.290612 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.292643 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.294893 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.296352 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.296488 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.296696 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.297007 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.297173 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11811,"thread_start_us":11639,"threads_started":1}
I20260929 02:55:58.297472 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.275276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:36383 (local address 127.0.0.1:58240)
0929 02:55:58.287531 (+ 12255us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:58.287617 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:58.287888 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:58.289906 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:58.289913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:58.289946 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:58.290412 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.290425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.292771 (+ 2346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.292776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.294702 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.294713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.296558 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.296564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.296745 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.296794 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 02:55:58.296937 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 02:55:58.297115 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11848,"thread_start_us":12257,"threads_started":1}
I20260929 02:55:58.298543 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.275480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:34653 (local address 127.0.0.1:56690)
0929 02:55:58.287821 (+ 12341us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:58.287839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:58.287966 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:58.291937 (+ 3971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:58.291942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:58.291968 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:58.292347 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.292357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.293984 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.293989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.296127 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.296141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.297759 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.297765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.297959 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.297998 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:55:58.298099 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 02:55:58.298241 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12217,"thread_start_us":12541,"threads_started":1}
I20260929 02:55:58.298929 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.275741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.15.18.67:34653)
0929 02:55:58.290944 (+ 15203us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.290953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.290986 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:55:58.291083 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.291091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.291345 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.291557 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:58.292494 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.293843 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.296283 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.297568 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.297693 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.297882 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.298179 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.298347 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15032,"thread_start_us":14884,"threads_started":1}
I20260929 02:55:58.299602 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.288435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:36045 (local address 127.0.0.1:44748)
0929 02:55:58.291800 (+ 3365us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:58.291816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:58.291912 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:58.292947 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:58.292952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:58.292974 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:58.293331 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.293340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.294939 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.294944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.296910 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.296922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.298876 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.298882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.299036 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.299066 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:55:58.299160 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:55:58.299278 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2653,"thread_start_us":3285,"threads_started":1}
I20260929 02:55:58.300028 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.288730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.15.18.66:36045)
0929 02:55:58.292420 (+ 3690us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.292429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.292453 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:55:58.292539 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.292545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.292769 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.292934 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:58.293498 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.294747 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.297071 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.298429 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.298554 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.298744 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.299607 (+ 863us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.299766 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3552,"thread_start_us":3327,"threads_started":1}
I20260929 02:55:58.301993 23151 tablet_service.cc:1467] Processing CreateTablet for tablet dcd68b111e2a48aba9f9a0807add2848 (DEFAULT_TABLE table=test-table [id=9072b95af67b46e28dcbf7b864192cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:58.302567 23429 tablet_service.cc:1467] Processing CreateTablet for tablet dcd68b111e2a48aba9f9a0807add2848 (DEFAULT_TABLE table=test-table [id=9072b95af67b46e28dcbf7b864192cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:58.303197 23151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd68b111e2a48aba9f9a0807add2848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:58.304047 23429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd68b111e2a48aba9f9a0807add2848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:58.303650 23290 tablet_service.cc:1467] Processing CreateTablet for tablet dcd68b111e2a48aba9f9a0807add2848 (DEFAULT_TABLE table=test-table [id=9072b95af67b46e28dcbf7b864192cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:58.304999 23290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd68b111e2a48aba9f9a0807add2848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:58.314675 23524 tablet_bootstrap.cc:492] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: Bootstrap starting.
I20260929 02:55:58.317951 23524 tablet_bootstrap.cc:654] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:58.319743 23526 tablet_bootstrap.cc:492] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: Bootstrap starting.
I20260929 02:55:58.320323 23524 log.cc:824] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:58.320508 23525 tablet_bootstrap.cc:492] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: Bootstrap starting.
I20260929 02:55:58.323151 23524 tablet_bootstrap.cc:492] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: No bootstrap required, opened a new log
I20260929 02:55:58.323418 23524 ts_tablet_manager.cc:1397] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 02:55:58.324033 23525 tablet_bootstrap.cc:654] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:58.324409 23526 tablet_bootstrap.cc:654] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:58.325413 23525 log.cc:824] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:58.325476 23526 log.cc:824] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:55:58.328147 23526 tablet_bootstrap.cc:492] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: No bootstrap required, opened a new log
I20260929 02:55:58.328393 23525 tablet_bootstrap.cc:492] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: No bootstrap required, opened a new log
I20260929 02:55:58.328460 23526 ts_tablet_manager.cc:1397] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:55:58.328701 23525 ts_tablet_manager.cc:1397] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 02:55:58.332696 23524 raft_consensus.cc:348] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } }
I20260929 02:55:58.333161 23524 raft_consensus.cc:374] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:58.333307 23524 raft_consensus.cc:729] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdef5fa80ed941cc8fd4932698bef4e5, State: Initialized, Role: FOLLOWER
I20260929 02:55:58.333959 23524 consensus_queue.cc:260] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } }
I20260929 02:55:58.335739 23524 ts_tablet_manager.cc:1428] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20260929 02:55:58.336423 23526 raft_consensus.cc:348] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } }
I20260929 02:55:58.336825 23526 raft_consensus.cc:374] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:58.336936 23526 raft_consensus.cc:729] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb1b689a4b54036a4ee37dfe35c1f7a, State: Initialized, Role: FOLLOWER
I20260929 02:55:58.337498 23526 consensus_queue.cc:260] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } }
I20260929 02:55:58.338122 23525 raft_consensus.cc:348] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } }
I20260929 02:55:58.338563 23525 raft_consensus.cc:374] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:58.338685 23525 raft_consensus.cc:729] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2cf3e1cb5841708d8a88efb52eacd3, State: Initialized, Role: FOLLOWER
I20260929 02:55:58.339171 23526 ts_tablet_manager.cc:1428] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260929 02:55:58.339248 23525 consensus_queue.cc:260] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } }
I20260929 02:55:58.341485 23525 ts_tablet_manager.cc:1428] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260929 02:55:58.454022 23361 tablet.cc:2369] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:55:58.456125 23222 tablet.cc:2369] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:55:58.461134 23500 tablet.cc:2369] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:55:58.471128 23531 raft_consensus.cc:482] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:58.471433 23531 raft_consensus.cc:504] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } }
I20260929 02:55:58.473272 23531 leader_election.cc:288] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383), ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045)
I20260929 02:55:58.480003 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.473676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:54803 (local address 127.15.18.65:36383)
0929 02:55:58.473891 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.473896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.474443 (+ 547us) server_negotiation.cc:407] Connection header received
0929 02:55:58.474691 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.474698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.474783 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.474877 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:58.475468 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.476293 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.478228 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.479067 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.479216 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.479302 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.479635 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.479759 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:55:58.480172 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.474746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:44641 (local address 127.15.18.66:36045)
0929 02:55:58.474943 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.474948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.475112 (+ 164us) server_negotiation.cc:407] Connection header received
0929 02:55:58.475301 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.475305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.475360 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.475431 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:58.475943 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.476664 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.478486 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.479291 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.479474 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.479582 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.479838 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.479959 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:55:58.480314 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.474471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:36045 (local address 127.15.18.67:44641)
0929 02:55:58.475028 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:58.475041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:58.475170 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:58.475462 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:58.475466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:58.475488 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:58.475808 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.475817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.476809 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.476812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.478353 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.478368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.479540 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.479546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.479702 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.479724 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:55:58.479802 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:55:58.479926 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":21,"thread_start_us":263,"threads_started":1}
I20260929 02:55:58.480693 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.473675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:36383 (local address 127.15.18.67:54803)
0929 02:55:58.474317 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:58.474341 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:58.474511 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:58.474976 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:58.474981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:58.475007 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:58.475331 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.475340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.476420 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.476425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.478066 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.478081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.479248 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.479253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.479412 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.479442 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:55:58.480183 (+ 741us) client_negotiation.cc:240] Negotiation successful
0929 02:55:58.480330 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":27,"spinlock_wait_cycles":43264,"thread_start_us":335,"threads_started":1}
I20260929 02:55:58.481412 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd68b111e2a48aba9f9a0807add2848" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" is_pre_election: true
I20260929 02:55:58.481748 23171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd68b111e2a48aba9f9a0807add2848" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" is_pre_election: true
I20260929 02:55:58.481886 23310 raft_consensus.cc:2454] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 0.
I20260929 02:55:58.482226 23171 raft_consensus.cc:2454] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 0.
I20260929 02:55:58.482493 23382 leader_election.cc:302] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec2cf3e1cb5841708d8a88efb52eacd3, efb1b689a4b54036a4ee37dfe35c1f7a; no voters:
I20260929 02:55:58.482836 23531 raft_consensus.cc:2781] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:58.482962 23531 raft_consensus.cc:482] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:58.483064 23531 raft_consensus.cc:3037] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:58.485451 23531 raft_consensus.cc:504] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } }
I20260929 02:55:58.485997 23531 leader_election.cc:288] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 election: Requested vote from peers fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383), ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045)
I20260929 02:55:58.486562 23171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd68b111e2a48aba9f9a0807add2848" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdef5fa80ed941cc8fd4932698bef4e5"
I20260929 02:55:58.486567 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd68b111e2a48aba9f9a0807add2848" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3"
I20260929 02:55:58.486747 23310 raft_consensus.cc:3037] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:58.486739 23171 raft_consensus.cc:3037] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:58.488940 23171 raft_consensus.cc:2454] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 1.
I20260929 02:55:58.488950 23310 raft_consensus.cc:2454] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 1.
I20260929 02:55:58.489377 23382 leader_election.cc:302] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efb1b689a4b54036a4ee37dfe35c1f7a, fdef5fa80ed941cc8fd4932698bef4e5; no voters:
I20260929 02:55:58.489662 23531 raft_consensus.cc:2781] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:58.490068 23531 raft_consensus.cc:686] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 LEADER]: Becoming Leader. State: Replica: efb1b689a4b54036a4ee37dfe35c1f7a, State: Running, Role: LEADER
I20260929 02:55:58.490567 23531 consensus_queue.cc:237] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } }
I20260929 02:55:58.495595 23015 catalog_manager.cc:5495] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a reported cstate change: term changed from 0 to 1, leader changed from <none> to efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67). New cstate: current_term: 1 leader_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } health_report { overall_health: UNKNOWN } } }
I20260929 02:55:58.539574 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.534015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.15.18.67:34653)
0929 02:55:58.534213 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.534219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.534329 (+ 110us) server_negotiation.cc:407] Connection header received
0929 02:55:58.534575 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.534580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.534668 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.534845 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:58.535223 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.536025 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.537324 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.537767 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.537875 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.538362 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:58.538927 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:58.539035 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.539191 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.539260 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:55:58.547679 23171 raft_consensus.cc:1261] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Refusing update from remote peer efb1b689a4b54036a4ee37dfe35c1f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:58.547683 23310 raft_consensus.cc:1261] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Refusing update from remote peer efb1b689a4b54036a4ee37dfe35c1f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:58.548619 23535 consensus_queue.cc:1035] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:58.548975 23531 consensus_queue.cc:1035] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:58.574316 23014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:58.574559 23014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:58.578025 23014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb: Sending DeleteTablet for 3 replicas of tablet dcd68b111e2a48aba9f9a0807add2848
I20260929 02:55:58.579157 23151 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd68b111e2a48aba9f9a0807add2848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:58 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:58240
I20260929 02:55:58.579480 23290 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd68b111e2a48aba9f9a0807add2848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:58 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:44748
I20260929 02:55:58.579871 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd68b111e2a48aba9f9a0807add2848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:58 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:56690
I20260929 02:55:58.580106 23548 tablet_replica.cc:330] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: stopping tablet replica
I20260929 02:55:58.580502 23548 raft_consensus.cc:2229] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:58.580979 23548 raft_consensus.cc:2258] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:58.581264 23014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:58.582149 23014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:58.582939 23548 ts_tablet_manager.cc:1905] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:58.589634 23548 ts_tablet_manager.cc:1918] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:58.589823 23548 log.cc:1196] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/dcd68b111e2a48aba9f9a0807add2848
I20260929 02:55:58.590341 23548 ts_tablet_manager.cc:1939] T dcd68b111e2a48aba9f9a0807add2848 P fdef5fa80ed941cc8fd4932698bef4e5: Deleting consensus metadata
I20260929 02:55:58.591491 23151 tablet_service.cc:1467] Processing CreateTablet for tablet 06eb426ea4c646f88914aa165bbd450f (DEFAULT_TABLE table=test-table [id=af5397afbc6c486ba959b2b9973499e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:58.591665 23289 tablet_service.cc:1467] Processing CreateTablet for tablet 06eb426ea4c646f88914aa165bbd450f (DEFAULT_TABLE table=test-table [id=af5397afbc6c486ba959b2b9973499e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:58.591965 23151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06eb426ea4c646f88914aa165bbd450f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:58.591965 23550 tablet_replica.cc:330] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: stopping tablet replica
I20260929 02:55:58.592176 23289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06eb426ea4c646f88914aa165bbd450f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:58.592396 23550 raft_consensus.cc:2229] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:58.592989 23550 raft_consensus.cc:2258] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:58.594859 23550 ts_tablet_manager.cc:1905] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:58.595855 23001 catalog_manager.cc:4841] TS fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383): tablet dcd68b111e2a48aba9f9a0807add2848 (table test-table [id=9072b95af67b46e28dcbf7b864192cf8]) successfully deleted
I20260929 02:55:58.596890 23428 tablet_service.cc:1467] Processing CreateTablet for tablet 06eb426ea4c646f88914aa165bbd450f (DEFAULT_TABLE table=test-table [id=af5397afbc6c486ba959b2b9973499e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:58.597400 23428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06eb426ea4c646f88914aa165bbd450f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:58.600873 23550 ts_tablet_manager.cc:1918] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:58.601022 23550 log.cc:1196] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/dcd68b111e2a48aba9f9a0807add2848
I20260929 02:55:58.601470 23550 ts_tablet_manager.cc:1939] T dcd68b111e2a48aba9f9a0807add2848 P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting consensus metadata
I20260929 02:55:58.603021 23000 catalog_manager.cc:4841] TS efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67:34653): tablet dcd68b111e2a48aba9f9a0807add2848 (table test-table [id=9072b95af67b46e28dcbf7b864192cf8]) successfully deleted
I20260929 02:55:58.603852 23525 tablet_bootstrap.cc:492] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3: Bootstrap starting.
I20260929 02:55:58.604142 23549 tablet_replica.cc:330] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: stopping tablet replica
I20260929 02:55:58.604666 23549 raft_consensus.cc:2229] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:58.605077 23524 tablet_bootstrap.cc:492] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5: Bootstrap starting.
I20260929 02:55:58.605333 23549 raft_consensus.cc:2258] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:58.605504 23525 tablet_bootstrap.cc:654] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:58.606580 23524 tablet_bootstrap.cc:654] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:58.607736 23549 ts_tablet_manager.cc:1905] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:58.614931 23549 ts_tablet_manager.cc:1918] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:58.615231 23526 tablet_bootstrap.cc:492] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a: Bootstrap starting.
I20260929 02:55:58.615233 23549 log.cc:1196] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/dcd68b111e2a48aba9f9a0807add2848
I20260929 02:55:58.615861 23549 ts_tablet_manager.cc:1939] T dcd68b111e2a48aba9f9a0807add2848 P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting consensus metadata
I20260929 02:55:58.616926 23526 tablet_bootstrap.cc:654] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:58.621883 23001 catalog_manager.cc:4841] TS ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045): tablet dcd68b111e2a48aba9f9a0807add2848 (table test-table [id=9072b95af67b46e28dcbf7b864192cf8]) successfully deleted
I20260929 02:55:58.625022 23525 tablet_bootstrap.cc:492] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3: No bootstrap required, opened a new log
I20260929 02:55:58.625202 23525 ts_tablet_manager.cc:1397] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.005s
I20260929 02:55:58.635684 23525 raft_consensus.cc:348] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.636421 23525 raft_consensus.cc:374] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:58.636546 23525 raft_consensus.cc:729] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2cf3e1cb5841708d8a88efb52eacd3, State: Initialized, Role: FOLLOWER
I20260929 02:55:58.638497 23525 consensus_queue.cc:260] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.644174 23525 ts_tablet_manager.cc:1428] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20260929 02:55:58.656021 23526 tablet_bootstrap.cc:492] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a: No bootstrap required, opened a new log
I20260929 02:55:58.656215 23526 ts_tablet_manager.cc:1397] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a: Time spent bootstrapping tablet: real 0.041s user 0.022s sys 0.016s
I20260929 02:55:58.656500 23524 tablet_bootstrap.cc:492] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5: No bootstrap required, opened a new log
I20260929 02:55:58.656658 23524 ts_tablet_manager.cc:1397] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5: Time spent bootstrapping tablet: real 0.052s user 0.019s sys 0.030s
I20260929 02:55:58.657099 23526 raft_consensus.cc:348] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.657342 23526 raft_consensus.cc:374] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:58.657439 23526 raft_consensus.cc:729] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb1b689a4b54036a4ee37dfe35c1f7a, State: Initialized, Role: FOLLOWER
I20260929 02:55:58.657395 23524 raft_consensus.cc:348] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.657612 23524 raft_consensus.cc:374] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:58.657691 23524 raft_consensus.cc:729] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdef5fa80ed941cc8fd4932698bef4e5, State: Initialized, Role: FOLLOWER
I20260929 02:55:58.657775 23526 consensus_queue.cc:260] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.657968 23524 consensus_queue.cc:260] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.658538 23524 ts_tablet_manager.cc:1428] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:55:58.660640 23526 ts_tablet_manager.cc:1428] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:55:58.730340 23530 raft_consensus.cc:482] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:58.730568 23530 raft_consensus.cc:504] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.732265 23530 leader_election.cc:288] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045), efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67:34653)
I20260929 02:55:58.740720 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.732651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:42349 (local address 127.15.18.66:36045)
0929 02:55:58.732851 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.732858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.733982 (+ 1124us) server_negotiation.cc:407] Connection header received
0929 02:55:58.734171 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.734177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.734253 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.734349 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:58.735110 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.736249 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.738511 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.739699 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.739868 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.739942 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.740307 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.740453 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:55:58.740897 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.732541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:36045 (local address 127.15.18.65:42349)
0929 02:55:58.733875 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:58.733894 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:58.734030 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:58.734408 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:58.734414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:58.734449 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:58.734869 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.734882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.736376 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.736381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.738352 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.738365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.739909 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.739916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.740087 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.740122 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:55:58.740269 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 02:55:58.740420 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"spinlock_wait_cycles":31488,"thread_start_us":1057,"threads_started":1}
I20260929 02:55:58.741202 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.733124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:45161 (local address 127.15.18.67:34653)
0929 02:55:58.733286 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.733292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.734523 (+ 1231us) server_negotiation.cc:407] Connection header received
0929 02:55:58.734729 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.734733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.734792 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.734877 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:55:58.735382 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.736345 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.738765 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.740010 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.740147 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.740216 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.740603 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.740836 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:55:58.741607 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06eb426ea4c646f88914aa165bbd450f" candidate_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" is_pre_election: true
I20260929 02:55:58.741828 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.732682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:34653 (local address 127.15.18.65:45161)
0929 02:55:58.734443 (+ 1761us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:58.734459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:58.734601 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:58.734903 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:58.734908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:58.734935 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:55:58.735245 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.735254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.736477 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.736483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.738613 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.738628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.740213 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.740219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.740415 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.740448 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:55:58.741319 (+ 871us) client_negotiation.cc:240] Negotiation successful
0929 02:55:58.741458 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1637,"spinlock_wait_cycles":4864,"thread_start_us":1430,"threads_started":1}
I20260929 02:55:58.741851 23310 raft_consensus.cc:2454] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdef5fa80ed941cc8fd4932698bef4e5 in term 0.
I20260929 02:55:58.742522 23104 leader_election.cc:302] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec2cf3e1cb5841708d8a88efb52eacd3, fdef5fa80ed941cc8fd4932698bef4e5; no voters:
I20260929 02:55:58.742717 23454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06eb426ea4c646f88914aa165bbd450f" candidate_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" is_pre_election: true
I20260929 02:55:58.743137 23454 raft_consensus.cc:2454] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdef5fa80ed941cc8fd4932698bef4e5 in term 0.
I20260929 02:55:58.743602 23530 raft_consensus.cc:2781] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:58.743759 23530 raft_consensus.cc:482] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:58.743862 23530 raft_consensus.cc:3037] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:58.745944 23530 raft_consensus.cc:504] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.746538 23530 leader_election.cc:288] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [CANDIDATE]: Term 1 election: Requested vote from peers ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045), efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67:34653)
I20260929 02:55:58.747087 23454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06eb426ea4c646f88914aa165bbd450f" candidate_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a"
I20260929 02:55:58.747288 23454 raft_consensus.cc:3037] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:58.749666 23454 raft_consensus.cc:2454] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdef5fa80ed941cc8fd4932698bef4e5 in term 1.
I20260929 02:55:58.750094 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06eb426ea4c646f88914aa165bbd450f" candidate_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3"
I20260929 02:55:58.750140 23103 leader_election.cc:302] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efb1b689a4b54036a4ee37dfe35c1f7a, fdef5fa80ed941cc8fd4932698bef4e5; no voters:
I20260929 02:55:58.750316 23310 raft_consensus.cc:3037] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:58.750473 23530 raft_consensus.cc:2781] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:58.751019 23530 raft_consensus.cc:686] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 LEADER]: Becoming Leader. State: Replica: fdef5fa80ed941cc8fd4932698bef4e5, State: Running, Role: LEADER
I20260929 02:55:58.751549 23530 consensus_queue.cc:237] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.753245 23310 raft_consensus.cc:2454] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdef5fa80ed941cc8fd4932698bef4e5 in term 1.
I20260929 02:55:58.755833 23015 catalog_manager.cc:5495] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65). New cstate: current_term: 1 leader_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:55:58 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650555, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.126@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:58.804056 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.789975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44301 (local address 127.0.0.1:35518)
0929 02:55:58.790800 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:58.790817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:58.790945 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:58.791299 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:58.791302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:58.792005 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:55:58.792254 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.792263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.793751 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.793755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.794685 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.794696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.795629 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.795634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.795764 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.796388 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:55:58.796405 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:55:58.798417 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:55:58.801760 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:58.801770 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:58.801776 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:58.802170 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:58.802506 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:55:58.802509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:55:58.802512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:55:58.802626 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:55:58.803161 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:55:58.803169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:55:58.803342 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 02:55:58.803489 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 02:55:58.803782 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":520,"threads_started":1}
I20260929 02:55:58.804306 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.789963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.15.18.126:44301)
0929 02:55:58.790595 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.790601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.790902 (+ 301us) server_negotiation.cc:407] Connection header received
0929 02:55:58.791063 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.791068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.791145 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.791265 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:58.792433 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.793632 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.794814 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.795432 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.795589 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.798593 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:58.798599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:58.798604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:58.798638 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:58.801638 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:58.802268 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:58.802274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:58.802277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:58.802414 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:58.802708 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:58.802713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:58.802716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:58.803009 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:58.803165 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.803602 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.803927 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":469,"thread_start_us":301,"threads_started":1}
I20260929 02:55:58.809090 23015 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:35518:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:55:58.810145 23015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:58.816754 23151 tablet_service.cc:1467] Processing CreateTablet for tablet cea29a69cf4c4db6b5d2f4a50341b3eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=538aa026030d4d84a1ecbf0cf6436781]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:55:58.816989 23428 tablet_service.cc:1467] Processing CreateTablet for tablet cea29a69cf4c4db6b5d2f4a50341b3eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=538aa026030d4d84a1ecbf0cf6436781]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:55:58.817360 23151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea29a69cf4c4db6b5d2f4a50341b3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:58.817503 23428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea29a69cf4c4db6b5d2f4a50341b3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:58.823202 23524 tablet_bootstrap.cc:492] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: Bootstrap starting.
I20260929 02:55:58.823712 23290 tablet_service.cc:1467] Processing CreateTablet for tablet cea29a69cf4c4db6b5d2f4a50341b3eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=538aa026030d4d84a1ecbf0cf6436781]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:55:58.824285 23290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea29a69cf4c4db6b5d2f4a50341b3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:58.824753 23526 tablet_bootstrap.cc:492] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: Bootstrap starting.
I20260929 02:55:58.825124 23524 tablet_bootstrap.cc:654] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:58.826442 23526 tablet_bootstrap.cc:654] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:58.833076 23525 tablet_bootstrap.cc:492] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: Bootstrap starting.
I20260929 02:55:58.834707 23525 tablet_bootstrap.cc:654] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:58.837910 23524 tablet_bootstrap.cc:492] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: No bootstrap required, opened a new log
I20260929 02:55:58.838104 23524 ts_tablet_manager.cc:1397] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.008s
I20260929 02:55:58.838165 23525 tablet_bootstrap.cc:492] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: No bootstrap required, opened a new log
I20260929 02:55:58.838514 23525 ts_tablet_manager.cc:1397] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:55:58.838889 23524 raft_consensus.cc:348] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.839145 23524 raft_consensus.cc:374] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:58.839228 23524 raft_consensus.cc:729] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdef5fa80ed941cc8fd4932698bef4e5, State: Initialized, Role: FOLLOWER
I20260929 02:55:58.839502 23524 consensus_queue.cc:260] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.839977 23530 tablet_replica.cc:439] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } }
I20260929 02:55:58.840256 23530 tablet_replica.cc:442] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:58.839987 23525 raft_consensus.cc:348] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.840538 23525 raft_consensus.cc:374] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:58.840703 23525 raft_consensus.cc:729] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2cf3e1cb5841708d8a88efb52eacd3, State: Initialized, Role: FOLLOWER
I20260929 02:55:58.842141 23524 ts_tablet_manager.cc:1428] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:55:58.842828 23525 consensus_queue.cc:260] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.843329 23532 tablet_replica.cc:439] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } }
I20260929 02:55:58.843729 23532 tablet_replica.cc:442] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:58.845263 23525 ts_tablet_manager.cc:1428] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260929 02:55:58.847129 23526 tablet_bootstrap.cc:492] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: No bootstrap required, opened a new log
I20260929 02:55:58.847314 23526 ts_tablet_manager.cc:1397] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.010s
I20260929 02:55:58.848155 23526 raft_consensus.cc:348] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.848402 23526 raft_consensus.cc:374] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:58.848481 23526 raft_consensus.cc:729] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb1b689a4b54036a4ee37dfe35c1f7a, State: Initialized, Role: FOLLOWER
I20260929 02:55:58.848731 23526 consensus_queue.cc:260] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.849164 23535 tablet_replica.cc:439] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } }
I20260929 02:55:58.849386 23526 ts_tablet_manager.cc:1428] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:55:58.849537 23535 tablet_replica.cc:442] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:58.892271 23535 raft_consensus.cc:482] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:58.892545 23535 raft_consensus.cc:504] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.893301 23535 leader_election.cc:288] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383), ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045)
I20260929 02:55:58.893882 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea29a69cf4c4db6b5d2f4a50341b3eb" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" is_pre_election: true
I20260929 02:55:58.893867 23171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea29a69cf4c4db6b5d2f4a50341b3eb" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" is_pre_election: true
I20260929 02:55:58.894124 23171 raft_consensus.cc:2454] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 0.
I20260929 02:55:58.894135 23310 raft_consensus.cc:2454] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 0.
I20260929 02:55:58.894572 23382 leader_election.cc:302] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efb1b689a4b54036a4ee37dfe35c1f7a, fdef5fa80ed941cc8fd4932698bef4e5; no voters:
I20260929 02:55:58.894898 23535 raft_consensus.cc:2781] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:58.895002 23535 raft_consensus.cc:482] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:58.895116 23535 raft_consensus.cc:3037] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:58.897450 23535 raft_consensus.cc:504] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.898002 23535 leader_election.cc:288] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 election: Requested vote from peers fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383), ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045)
I20260929 02:55:58.898840 23171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea29a69cf4c4db6b5d2f4a50341b3eb" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdef5fa80ed941cc8fd4932698bef4e5"
I20260929 02:55:58.899070 23171 raft_consensus.cc:3037] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:58.899152 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea29a69cf4c4db6b5d2f4a50341b3eb" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3"
I20260929 02:55:58.899410 23310 raft_consensus.cc:3037] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:58.901264 23171 raft_consensus.cc:2454] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 1.
I20260929 02:55:58.901710 23382 leader_election.cc:302] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efb1b689a4b54036a4ee37dfe35c1f7a, fdef5fa80ed941cc8fd4932698bef4e5; no voters:
I20260929 02:55:58.901974 23535 raft_consensus.cc:2781] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:58.902320 23310 raft_consensus.cc:2454] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 1.
I20260929 02:55:58.902143 23535 raft_consensus.cc:686] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 LEADER]: Becoming Leader. State: Replica: efb1b689a4b54036a4ee37dfe35c1f7a, State: Running, Role: LEADER
I20260929 02:55:58.902617 23535 consensus_queue.cc:237] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:58.910955 23531 tablet_replica.cc:439] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: TxnStatusTablet state changed. Reason: New leader efb1b689a4b54036a4ee37dfe35c1f7a. Latest consensus state: current_term: 1 leader_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } }
I20260929 02:55:58.911273 23531 tablet_replica.cc:442] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:55:58.912063 23569 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:55:58.912294 23569 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:55:58.912262 23015 catalog_manager.cc:5495] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a reported cstate change: term changed from 0 to 1, leader changed from <none> to efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67). New cstate: current_term: 1 leader_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } health_report { overall_health: HEALTHY } } }
I20260929 02:55:58.936551 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.931460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.15.18.67:34653)
0929 02:55:58.931665 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.931670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.931692 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:55:58.931992 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.931996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.932061 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.932133 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:58.932490 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.933260 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.934632 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.935234 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.935353 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.935802 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:58.935920 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:58.935994 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.936133 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.936213 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 02:55:58.936535 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.931317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:34653 (local address 127.0.0.1:56710)
0929 02:55:58.931625 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:58.931644 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:58.931774 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:58.932173 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:58.932177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:58.932189 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:58.932361 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.932369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.933392 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.933398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.934478 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.934491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.935379 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.935385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.935573 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.935642 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:58.936010 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:58.936030 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 02:55:58.936103 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 02:55:58.936253 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 02:55:58.940719 23171 raft_consensus.cc:1261] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Refusing update from remote peer efb1b689a4b54036a4ee37dfe35c1f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:58.940745 23310 raft_consensus.cc:1261] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Refusing update from remote peer efb1b689a4b54036a4ee37dfe35c1f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:58.941337 23535 consensus_queue.cc:1035] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:58.941708 23531 consensus_queue.cc:1035] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:58.945855 23532 tablet_replica.cc:439] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: TxnStatusTablet state changed. Reason: New leader efb1b689a4b54036a4ee37dfe35c1f7a. Latest consensus state: current_term: 1 leader_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } }
I20260929 02:55:58.946182 23532 tablet_replica.cc:442] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:58.948295 23531 tablet_replica.cc:439] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } }
I20260929 02:55:58.948591 23531 tablet_replica.cc:442] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:55:58.949955 23538 mvcc.cc:205] Tried to move back new op lower bound from 7334504689415192576 to 7334504689268011008. Current Snapshot: MvccSnapshot[applied={T|T < 7334504689415192576 or (T in {7334504689415192576})}]
I20260929 02:55:58.951386 23540 mvcc.cc:205] Tried to move back new op lower bound from 7334504689415192576 to 7334504689268011008. Current Snapshot: MvccSnapshot[applied={T|T < 7334504689415192576}]
I20260929 02:55:58.951738 23532 tablet_replica.cc:439] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } }
I20260929 02:55:58.951977 23532 tablet_replica.cc:442] T cea29a69cf4c4db6b5d2f4a50341b3eb P ec2cf3e1cb5841708d8a88efb52eacd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:58.957808 23530 tablet_replica.cc:439] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: TxnStatusTablet state changed. Reason: New leader efb1b689a4b54036a4ee37dfe35c1f7a. Latest consensus state: current_term: 1 leader_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } }
I20260929 02:55:58.958137 23530 tablet_replica.cc:442] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:58.959384 23543 tablet_replica.cc:439] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } }
I20260929 02:55:58.959798 23543 tablet_replica.cc:442] T cea29a69cf4c4db6b5d2f4a50341b3eb P efb1b689a4b54036a4ee37dfe35c1f7a: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:55:58.961047 23530 tablet_replica.cc:439] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } } }
I20260929 02:55:58.961344 23530 tablet_replica.cc:442] T cea29a69cf4c4db6b5d2f4a50341b3eb P fdef5fa80ed941cc8fd4932698bef4e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:55:58.967443 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.960646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.15.18.65:36383)
0929 02:55:58.960813 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.960819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.961267 (+ 448us) server_negotiation.cc:407] Connection header received
0929 02:55:58.961377 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.961383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.961479 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.961629 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:58.962215 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.963292 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.965219 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.965617 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.965744 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.966216 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:58.966782 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:58.966893 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.967078 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.967152 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:55:58.988801 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.978001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:60493 (local address 127.15.18.67:34653)
0929 02:55:58.978233 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:58.978240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:58.983947 (+ 5707us) server_negotiation.cc:407] Connection header received
0929 02:55:58.984145 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:58.984150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:58.984221 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:58.984307 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:58.984838 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.985576 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.986957 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.987426 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.987569 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.988044 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:58.988165 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:58.988248 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:58.988382 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:55:58.988469 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:55:58.988873 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:58.977964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:34653 (local address 127.15.18.65:60493)
0929 02:55:58.983829 (+ 5865us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:58.983848 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:58.984015 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:58.984356 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:58.984361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:58.984373 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:58.984657 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.984667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.985717 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.985721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.986807 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:58.986822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:58.987640 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:58.987646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:58.987777 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:58.987851 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:58.988262 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:58.988274 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 02:55:58.988346 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 02:55:58.988499 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5683,"thread_start_us":5525,"threads_started":1}
I20260929 02:55:58.999796 23310 raft_consensus.cc:1261] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Refusing update from remote peer fdef5fa80ed941cc8fd4932698bef4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:58.999998 23454 raft_consensus.cc:1261] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Refusing update from remote peer fdef5fa80ed941cc8fd4932698bef4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:59.000514 23556 consensus_queue.cc:1035] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:59.000936 23530 consensus_queue.cc:1035] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:59.009445 23539 mvcc.cc:205] Tried to move back new op lower bound from 7334504689655439360 to 7334504688650932224. Current Snapshot: MvccSnapshot[applied={T|T < 7334504689655439360}]
I20260929 02:55:59.043804 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:59.038233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:58645 (local address 127.15.18.65:36383)
0929 02:55:59.038416 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:59.038422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:59.038463 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:55:59.038628 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:59.038633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:59.038711 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:59.038798 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:59.039285 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.040394 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.041983 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.042524 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.042638 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.043084 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:59.043192 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:59.043264 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:59.043418 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 02:55:59.043495 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:55:59.044221 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:59.037720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:36383 (local address 127.15.18.67:58645)
0929 02:55:59.038387 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:59.038404 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:59.038524 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:59.038805 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:59.038808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:59.038815 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:59.039136 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:59.039149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.040521 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.040525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:59.041852 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:59.041865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.042685 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.042692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:59.042863 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.042923 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:59.043295 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:59.043309 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 02:55:59.043400 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:55:59.043625 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":316,"threads_started":1}
I20260929 02:55:59.084139 23015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:59.084430 23015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:59.088078 23015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb: Sending DeleteTablet for 3 replicas of tablet 06eb426ea4c646f88914aa165bbd450f
I20260929 02:55:59.089279 23290 tablet_service.cc:1514] Processing DeleteTablet for tablet 06eb426ea4c646f88914aa165bbd450f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:59 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:44748
I20260929 02:55:59.089607 23549 tablet_replica.cc:330] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3: stopping tablet replica
I20260929 02:55:59.089849 23549 raft_consensus.cc:2229] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:59.090045 23549 raft_consensus.cc:2258] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:59.090533 23151 tablet_service.cc:1514] Processing DeleteTablet for tablet 06eb426ea4c646f88914aa165bbd450f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:59 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:58240
I20260929 02:55:59.090847 23548 tablet_replica.cc:330] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5: stopping tablet replica
I20260929 02:55:59.091058 23548 raft_consensus.cc:2229] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:59.091619 23548 raft_consensus.cc:2258] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:59.093906 23549 ts_tablet_manager.cc:1905] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:59.095861 23428 tablet_service.cc:1514] Processing DeleteTablet for tablet 06eb426ea4c646f88914aa165bbd450f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:59 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:56690
I20260929 02:55:59.096151 23550 tablet_replica.cc:330] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a: stopping tablet replica
I20260929 02:55:59.096355 23550 raft_consensus.cc:2229] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:59.096555 23550 raft_consensus.cc:2258] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:59.098079 23548 ts_tablet_manager.cc:1905] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:59.104573 23550 ts_tablet_manager.cc:1905] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:59.109548 23548 ts_tablet_manager.cc:1918] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:59.109754 23548 log.cc:1196] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/06eb426ea4c646f88914aa165bbd450f
I20260929 02:55:59.110230 23548 ts_tablet_manager.cc:1939] T 06eb426ea4c646f88914aa165bbd450f P fdef5fa80ed941cc8fd4932698bef4e5: Deleting consensus metadata
I20260929 02:55:59.111464 23001 catalog_manager.cc:4841] TS fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383): tablet 06eb426ea4c646f88914aa165bbd450f (table test-table [id=af5397afbc6c486ba959b2b9973499e2]) successfully deleted
I20260929 02:55:59.117313 23550 ts_tablet_manager.cc:1918] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:59.117935 23549 ts_tablet_manager.cc:1918] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:59.118220 23550 log.cc:1196] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/06eb426ea4c646f88914aa165bbd450f
I20260929 02:55:59.118412 23549 log.cc:1196] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/06eb426ea4c646f88914aa165bbd450f
I20260929 02:55:59.118888 23550 ts_tablet_manager.cc:1939] T 06eb426ea4c646f88914aa165bbd450f P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting consensus metadata
I20260929 02:55:59.119386 23549 ts_tablet_manager.cc:1939] T 06eb426ea4c646f88914aa165bbd450f P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting consensus metadata
I20260929 02:55:59.127658 23001 catalog_manager.cc:4841] TS ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045): tablet 06eb426ea4c646f88914aa165bbd450f (table test-table [id=af5397afbc6c486ba959b2b9973499e2]) successfully deleted
I20260929 02:55:59.128024 23000 catalog_manager.cc:4841] TS efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67:34653): tablet 06eb426ea4c646f88914aa165bbd450f (table test-table [id=af5397afbc6c486ba959b2b9973499e2]) successfully deleted
Sep 29 02:55:59 dist-test-slave-q5f0 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:55:59 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:55:59 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:55:59.167673 23563 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:55:59.153985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.15.18.126:44301)
0929 02:55:59.154205 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:59.154224 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:59.154612 (+ 388us) server_negotiation.cc:407] Connection header received
0929 02:55:59.154826 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:59.154832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:59.154915 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:59.155017 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:59.156689 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.157845 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.159774 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.160447 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.160582 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.167332 (+ 6750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35528: BlockingRecv error: recv got EOF from 127.0.0.1:35528 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 02:55:59 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM
I20260929 02:55:59.198099 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:59.176937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.15.18.126:44301)
0929 02:55:59.177132 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:59.177139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:59.180146 (+ 3007us) server_negotiation.cc:407] Connection header received
0929 02:55:59.180326 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:59.180332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:59.180417 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:59.180528 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:59.182116 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.183280 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.185230 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.185839 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.185965 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.190827 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:59.190834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:59.190840 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:59.190887 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:59.194407 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:59.195257 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:59.195264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:59.195267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:59.195420 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:59.195905 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:59.195911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:59.195914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:59.196299 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:59.196566 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:59.197481 (+ 915us) server_negotiation.cc:299] Negotiation successful
0929 02:55:59.197732 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:55:59.205456 23015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:55:59.206140 23015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:59.215057 23151 tablet_service.cc:1467] Processing CreateTablet for tablet bb6a490bd8184372b43d9e8dceecc3cc (DEFAULT_TABLE table=test-table [id=60c4b0aada7a4d6088a81e2f59fca79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:59.215540 23151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6a490bd8184372b43d9e8dceecc3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:59.219945 23290 tablet_service.cc:1467] Processing CreateTablet for tablet bb6a490bd8184372b43d9e8dceecc3cc (DEFAULT_TABLE table=test-table [id=60c4b0aada7a4d6088a81e2f59fca79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:59.220397 23290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6a490bd8184372b43d9e8dceecc3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:59.223914 23428 tablet_service.cc:1467] Processing CreateTablet for tablet bb6a490bd8184372b43d9e8dceecc3cc (DEFAULT_TABLE table=test-table [id=60c4b0aada7a4d6088a81e2f59fca79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:59.224362 23428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6a490bd8184372b43d9e8dceecc3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:59.238736 23525 tablet_bootstrap.cc:492] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: Bootstrap starting.
I20260929 02:55:59.240552 23525 tablet_bootstrap.cc:654] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:59.242576 23526 tablet_bootstrap.cc:492] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a: Bootstrap starting.
I20260929 02:55:59.244333 23526 tablet_bootstrap.cc:654] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:59.247562 23524 tablet_bootstrap.cc:492] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5: Bootstrap starting.
I20260929 02:55:59.249208 23524 tablet_bootstrap.cc:654] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:59.252750 23524 tablet_bootstrap.cc:492] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5: No bootstrap required, opened a new log
I20260929 02:55:59.252938 23524 ts_tablet_manager.cc:1397] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 02:55:59.253774 23524 raft_consensus.cc:348] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.254055 23524 raft_consensus.cc:374] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:59.254150 23524 raft_consensus.cc:729] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdef5fa80ed941cc8fd4932698bef4e5, State: Initialized, Role: FOLLOWER
I20260929 02:55:59.254428 23524 consensus_queue.cc:260] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.258596 23524 ts_tablet_manager.cc:1428] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20260929 02:55:59.259178 23526 tablet_bootstrap.cc:492] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a: No bootstrap required, opened a new log
I20260929 02:55:59.259414 23526 ts_tablet_manager.cc:1397] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20260929 02:55:59.260396 23526 raft_consensus.cc:348] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.260821 23526 raft_consensus.cc:374] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:59.260977 23526 raft_consensus.cc:729] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb1b689a4b54036a4ee37dfe35c1f7a, State: Initialized, Role: FOLLOWER
I20260929 02:55:59.261344 23526 consensus_queue.cc:260] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.261602 23525 tablet_bootstrap.cc:492] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: No bootstrap required, opened a new log
I20260929 02:55:59.261766 23525 ts_tablet_manager.cc:1397] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20260929 02:55:59.262140 23526 ts_tablet_manager.cc:1428] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:55:59.262523 23525 raft_consensus.cc:348] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.262780 23525 raft_consensus.cc:374] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:59.262871 23525 raft_consensus.cc:729] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2cf3e1cb5841708d8a88efb52eacd3, State: Initialized, Role: FOLLOWER
I20260929 02:55:59.263470 23525 consensus_queue.cc:260] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.264019 23525 ts_tablet_manager.cc:1428] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:59.284698 23530 raft_consensus.cc:482] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:59.284897 23530 raft_consensus.cc:504] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.285636 23530 leader_election.cc:288] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045), efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67:34653)
I20260929 02:55:59.286109 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6a490bd8184372b43d9e8dceecc3cc" candidate_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" is_pre_election: true
I20260929 02:55:59.286374 23310 raft_consensus.cc:2454] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdef5fa80ed941cc8fd4932698bef4e5 in term 0.
I20260929 02:55:59.286754 23454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6a490bd8184372b43d9e8dceecc3cc" candidate_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" is_pre_election: true
I20260929 02:55:59.286861 23104 leader_election.cc:302] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec2cf3e1cb5841708d8a88efb52eacd3, fdef5fa80ed941cc8fd4932698bef4e5; no voters:
I20260929 02:55:59.286998 23454 raft_consensus.cc:2454] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdef5fa80ed941cc8fd4932698bef4e5 in term 0.
I20260929 02:55:59.287154 23530 raft_consensus.cc:2781] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:59.287281 23530 raft_consensus.cc:482] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:59.287374 23530 raft_consensus.cc:3037] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:59.290720 23530 raft_consensus.cc:504] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.291419 23530 leader_election.cc:288] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [CANDIDATE]: Term 1 election: Requested vote from peers ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045), efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67:34653)
I20260929 02:55:59.292196 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6a490bd8184372b43d9e8dceecc3cc" candidate_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3"
I20260929 02:55:59.292418 23310 raft_consensus.cc:3037] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:59.295476 23310 raft_consensus.cc:2454] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdef5fa80ed941cc8fd4932698bef4e5 in term 1.
I20260929 02:55:59.295928 23104 leader_election.cc:302] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec2cf3e1cb5841708d8a88efb52eacd3, fdef5fa80ed941cc8fd4932698bef4e5; no voters:
I20260929 02:55:59.296064 23454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6a490bd8184372b43d9e8dceecc3cc" candidate_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a"
I20260929 02:55:59.296308 23454 raft_consensus.cc:3037] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:59.296357 23530 raft_consensus.cc:2781] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:59.296900 23530 raft_consensus.cc:686] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 LEADER]: Becoming Leader. State: Replica: fdef5fa80ed941cc8fd4932698bef4e5, State: Running, Role: LEADER
I20260929 02:55:59.297240 23530 consensus_queue.cc:237] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.299480 23454 raft_consensus.cc:2454] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdef5fa80ed941cc8fd4932698bef4e5 in term 1.
I20260929 02:55:59.302071 23014 catalog_manager.cc:5495] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65). New cstate: current_term: 1 leader_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } health_report { overall_health: UNKNOWN } } }
I20260929 02:55:59.320456 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:59.313857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.15.18.65:36383)
0929 02:55:59.314512 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:59.314518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:59.314534 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:55:59.314597 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:59.314606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:59.314694 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:59.314777 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:59.315196 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.316349 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.318322 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.319006 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.319131 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.319661 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:59.319821 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:59.319910 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:59.320042 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 02:55:59.320131 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512}
I20260929 02:55:59.323737 23310 raft_consensus.cc:1261] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Refusing update from remote peer fdef5fa80ed941cc8fd4932698bef4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:59.323750 23454 raft_consensus.cc:1261] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Refusing update from remote peer fdef5fa80ed941cc8fd4932698bef4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:59.324375 23579 consensus_queue.cc:1035] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:59.324926 23530 consensus_queue.cc:1035] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:59.339501 23015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:59.339793 23015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:59.343896 23015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb: Sending DeleteTablet for 3 replicas of tablet bb6a490bd8184372b43d9e8dceecc3cc
I20260929 02:55:59.344822 23428 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6a490bd8184372b43d9e8dceecc3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:59 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:56690
I20260929 02:55:59.345114 23550 tablet_replica.cc:330] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a: stopping tablet replica
I20260929 02:55:59.345293 23550 raft_consensus.cc:2229] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:59.345523 23550 raft_consensus.cc:2258] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:59.346421 23290 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6a490bd8184372b43d9e8dceecc3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:59 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:44748
I20260929 02:55:59.346719 23549 tablet_replica.cc:330] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: stopping tablet replica
I20260929 02:55:59.347195 23151 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6a490bd8184372b43d9e8dceecc3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:59 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:58240
I20260929 02:55:59.347476 23548 tablet_replica.cc:330] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5: stopping tablet replica
I20260929 02:55:59.347494 23550 ts_tablet_manager.cc:1905] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:59.347666 23548 raft_consensus.cc:2229] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:59.347958 23548 raft_consensus.cc:2258] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:59.348963 23548 ts_tablet_manager.cc:1905] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:59.354619 23550 ts_tablet_manager.cc:1918] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:59.354799 23550 log.cc:1196] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/bb6a490bd8184372b43d9e8dceecc3cc
I20260929 02:55:59.354934 23015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 02:55:59.355276 23550 ts_tablet_manager.cc:1939] T bb6a490bd8184372b43d9e8dceecc3cc P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting consensus metadata
W20260929 02:55:59.355871 23015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:55:59.356534 23000 catalog_manager.cc:4841] TS efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67:34653): tablet bb6a490bd8184372b43d9e8dceecc3cc (table test-table [id=60c4b0aada7a4d6088a81e2f59fca79b]) successfully deleted
I20260929 02:55:59.360379 23549 raft_consensus.cc:2229] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:59.360921 23549 pending_rounds.cc:62] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: Trying to abort 2 pending ops.
I20260929 02:55:59.361044 23549 pending_rounds.cc:68] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334504690982985728 op_type: WRITE_OP write_request { tablet_id: "bb6a490bd8184372b43d9e8dceecc3cc" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\353\323\354\325\006\"8\n\ntest-admin\022*\n 60c4b0aada7a4d6088a81e2f59fca79b\020\001\030\001 \001(\001" signature: "<redacted>""\200\223\317\tH\037D1\270\014A\006\272\350\233\337\305\212&;\344E\361|\307^<G\357\255Q\250\220Y\033!:tC\234\\+\374:\262\373c\217\022]\225c\306w\301\r\356\014\347\363\3315\240\355" signing_key_seq_num: 0 } } request_id { client_id: "3f721a3e9fd14d018fc2aad2cbcae4f5" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260929 02:55:59.361730 23549 pending_rounds.cc:68] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334504690884427776 op_type: NO_OP noop_request { }
I20260929 02:55:59.361878 23549 raft_consensus.cc:2866] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260929 02:55:59.362027 23549 raft_consensus.cc:2258] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:59.362327 23548 ts_tablet_manager.cc:1918] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:59.362496 23548 log.cc:1196] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/bb6a490bd8184372b43d9e8dceecc3cc
I20260929 02:55:59.362937 23548 ts_tablet_manager.cc:1939] T bb6a490bd8184372b43d9e8dceecc3cc P fdef5fa80ed941cc8fd4932698bef4e5: Deleting consensus metadata
I20260929 02:55:59.364200 23001 catalog_manager.cc:4841] TS fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383): tablet bb6a490bd8184372b43d9e8dceecc3cc (table test-table [id=60c4b0aada7a4d6088a81e2f59fca79b]) successfully deleted
I20260929 02:55:59.366885 23151 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6a490bd8184372b43d9e8dceecc3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:59 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:58240
I20260929 02:55:59.368083 23549 ts_tablet_manager.cc:1905] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:59.371745 23428 tablet_service.cc:1467] Processing CreateTablet for tablet c9fe6d83e8c74ca9b15c45834bdc6a6d (DEFAULT_TABLE table=test-table [id=6258bb881956487d99b7f9a94e92e2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:59.372224 23428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fe6d83e8c74ca9b15c45834bdc6a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:59.374866 23549 ts_tablet_manager.cc:1918] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:55:59.374998 23549 log.cc:1196] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/bb6a490bd8184372b43d9e8dceecc3cc
I20260929 02:55:59.375392 23549 ts_tablet_manager.cc:1939] T bb6a490bd8184372b43d9e8dceecc3cc P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting consensus metadata
I20260929 02:55:59.376480 23290 tablet_service.cc:1467] Processing CreateTablet for tablet c9fe6d83e8c74ca9b15c45834bdc6a6d (DEFAULT_TABLE table=test-table [id=6258bb881956487d99b7f9a94e92e2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:59.376912 23290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fe6d83e8c74ca9b15c45834bdc6a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:59.377199 23001 catalog_manager.cc:4841] TS ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045): tablet bb6a490bd8184372b43d9e8dceecc3cc (table test-table [id=60c4b0aada7a4d6088a81e2f59fca79b]) successfully deleted
I20260929 02:55:59.380003 23151 tablet_service.cc:1467] Processing CreateTablet for tablet c9fe6d83e8c74ca9b15c45834bdc6a6d (DEFAULT_TABLE table=test-table [id=6258bb881956487d99b7f9a94e92e2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:55:59.380501 23151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fe6d83e8c74ca9b15c45834bdc6a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:55:59.382871 23526 tablet_bootstrap.cc:492] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a: Bootstrap starting.
W20260929 02:55:59.383721 23001 catalog_manager.cc:4808] TS fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383): delete failed for tablet bb6a490bd8184372b43d9e8dceecc3cc because the tablet was not found. No further retry: Not found: Tablet not found: bb6a490bd8184372b43d9e8dceecc3cc
I20260929 02:55:59.384706 23526 tablet_bootstrap.cc:654] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:59.396498 23525 tablet_bootstrap.cc:492] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3: Bootstrap starting.
I20260929 02:55:59.398277 23525 tablet_bootstrap.cc:654] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:59.399044 23526 tablet_bootstrap.cc:492] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a: No bootstrap required, opened a new log
I20260929 02:55:59.399230 23526 ts_tablet_manager.cc:1397] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20260929 02:55:59.399400 23524 tablet_bootstrap.cc:492] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5: Bootstrap starting.
I20260929 02:55:59.399926 23526 raft_consensus.cc:348] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.400134 23526 raft_consensus.cc:374] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:59.400204 23526 raft_consensus.cc:729] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb1b689a4b54036a4ee37dfe35c1f7a, State: Initialized, Role: FOLLOWER
I20260929 02:55:59.400471 23526 consensus_queue.cc:260] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.401038 23526 ts_tablet_manager.cc:1428] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:59.401109 23524 tablet_bootstrap.cc:654] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5: Neither blocks nor log segments found. Creating new log.
I20260929 02:55:59.407301 23525 tablet_bootstrap.cc:492] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3: No bootstrap required, opened a new log
I20260929 02:55:59.407492 23525 ts_tablet_manager.cc:1397] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260929 02:55:59.408330 23525 raft_consensus.cc:348] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.408595 23525 raft_consensus.cc:374] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:59.408677 23525 raft_consensus.cc:729] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2cf3e1cb5841708d8a88efb52eacd3, State: Initialized, Role: FOLLOWER
I20260929 02:55:59.410053 23525 consensus_queue.cc:260] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.412777 23525 ts_tablet_manager.cc:1428] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 02:55:59.415719 23524 tablet_bootstrap.cc:492] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5: No bootstrap required, opened a new log
I20260929 02:55:59.415917 23524 ts_tablet_manager.cc:1397] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20260929 02:55:59.416780 23524 raft_consensus.cc:348] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.417078 23524 raft_consensus.cc:374] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:55:59.417165 23524 raft_consensus.cc:729] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdef5fa80ed941cc8fd4932698bef4e5, State: Initialized, Role: FOLLOWER
I20260929 02:55:59.417424 23524 consensus_queue.cc:260] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.418074 23524 ts_tablet_manager.cc:1428] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:55:59.569109 23535 raft_consensus.cc:482] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:55:59.569327 23535 raft_consensus.cc:504] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.569952 23535 leader_election.cc:288] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045), fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383)
I20260929 02:55:59.570578 23171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe6d83e8c74ca9b15c45834bdc6a6d" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" is_pre_election: true
I20260929 02:55:59.570616 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe6d83e8c74ca9b15c45834bdc6a6d" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" is_pre_election: true
I20260929 02:55:59.570878 23171 raft_consensus.cc:2454] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 0.
I20260929 02:55:59.570911 23310 raft_consensus.cc:2454] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 0.
I20260929 02:55:59.571532 23382 leader_election.cc:302] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec2cf3e1cb5841708d8a88efb52eacd3, efb1b689a4b54036a4ee37dfe35c1f7a; no voters:
I20260929 02:55:59.571913 23535 raft_consensus.cc:2781] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:55:59.572033 23535 raft_consensus.cc:482] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:55:59.572124 23535 raft_consensus.cc:3037] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:59.574049 23535 raft_consensus.cc:504] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.574559 23535 leader_election.cc:288] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 election: Requested vote from peers ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045), fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383)
I20260929 02:55:59.575119 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe6d83e8c74ca9b15c45834bdc6a6d" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3"
I20260929 02:55:59.575238 23171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fe6d83e8c74ca9b15c45834bdc6a6d" candidate_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdef5fa80ed941cc8fd4932698bef4e5"
I20260929 02:55:59.575361 23310 raft_consensus.cc:3037] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:59.575459 23171 raft_consensus.cc:3037] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:55:59.578297 23310 raft_consensus.cc:2454] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 1.
I20260929 02:55:59.578467 23171 raft_consensus.cc:2454] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb1b689a4b54036a4ee37dfe35c1f7a in term 1.
I20260929 02:55:59.578831 23382 leader_election.cc:302] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec2cf3e1cb5841708d8a88efb52eacd3, efb1b689a4b54036a4ee37dfe35c1f7a; no voters:
I20260929 02:55:59.579126 23535 raft_consensus.cc:2781] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:55:59.579538 23535 raft_consensus.cc:686] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 LEADER]: Becoming Leader. State: Replica: efb1b689a4b54036a4ee37dfe35c1f7a, State: Running, Role: LEADER
I20260929 02:55:59.579811 23535 consensus_queue.cc:237] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } } peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } }
I20260929 02:55:59.582386 23015 catalog_manager.cc:5495] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a reported cstate change: term changed from 0 to 1, leader changed from <none> to efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67). New cstate: current_term: 1 leader_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb1b689a4b54036a4ee37dfe35c1f7a" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 34653 } health_report { overall_health: HEALTHY } } }
I20260929 02:55:59.616585 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:59.608438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.15.18.67:34653)
0929 02:55:59.611968 (+ 3530us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:59.611972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:59.611988 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:55:59.612043 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:59.612046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:59.612108 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:59.612209 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:59.612621 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.613318 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.614789 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.615222 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.615340 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.615842 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:59.615972 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:59.616050 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:59.616192 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 02:55:59.616279 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3429,"thread_start_us":3318,"threads_started":1}
I20260929 02:55:59.631345 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:59.625210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:39623 (local address 127.15.18.67:34653)
0929 02:55:59.625387 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:59.625392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:59.626013 (+ 621us) server_negotiation.cc:407] Connection header received
0929 02:55:59.626181 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:59.626185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:59.626254 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:59.626329 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:55:59.626790 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.627486 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.629533 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.630042 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.630163 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.630703 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:55:59.630818 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:55:59.630914 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:59.631028 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 02:55:59.631095 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:55:59.631593 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:59.625186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:34653 (local address 127.15.18.67:39623)
0929 02:55:59.625892 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 02:55:59.625911 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:55:59.626065 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:55:59.626355 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:55:59.626360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:55:59.626372 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:55:59.626647 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:59.626659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.627636 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.627641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:59.629383 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:55:59.629396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.630226 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.630232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:55:59.630463 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.630510 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:55:59.630914 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:55:59.630924 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:55:59.631003 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:55:59.631156 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":3,"thread_start_us":417,"threads_started":1}
I20260929 02:55:59.640087 23310 raft_consensus.cc:1261] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Refusing update from remote peer efb1b689a4b54036a4ee37dfe35c1f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:59.640093 23171 raft_consensus.cc:1261] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Refusing update from remote peer efb1b689a4b54036a4ee37dfe35c1f7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:55:59.640693 23535 consensus_queue.cc:1035] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdef5fa80ed941cc8fd4932698bef4e5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:59.640955 23606 consensus_queue.cc:1035] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec2cf3e1cb5841708d8a88efb52eacd3" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:55:59.690954 23013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:59.691181 23013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:55:59.694104 23013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf57cd62707e49dbbb54e123056e9cfb: Sending DeleteTablet for 3 replicas of tablet c9fe6d83e8c74ca9b15c45834bdc6a6d
I20260929 02:55:59.694977 23290 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fe6d83e8c74ca9b15c45834bdc6a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:59 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:44748
I20260929 02:55:59.695026 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fe6d83e8c74ca9b15c45834bdc6a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:59 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:56690
I20260929 02:55:59.695302 23550 tablet_replica.cc:330] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a: stopping tablet replica
I20260929 02:55:59.695312 23549 tablet_replica.cc:330] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3: stopping tablet replica
I20260929 02:55:59.695487 23550 raft_consensus.cc:2229] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:55:59.695504 23549 raft_consensus.cc:2229] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:59.695729 23549 raft_consensus.cc:2258] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:59.695799 23550 raft_consensus.cc:2258] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:59.696087 23151 tablet_service.cc:1514] Processing DeleteTablet for tablet c9fe6d83e8c74ca9b15c45834bdc6a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:55:59 UTC) from {username='oryx', principal='oryx/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:58240
I20260929 02:55:59.696452 23548 tablet_replica.cc:330] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5: stopping tablet replica
I20260929 02:55:59.696683 23548 raft_consensus.cc:2229] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:55:59.696875 23548 raft_consensus.cc:2258] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:55:59.696882 23549 ts_tablet_manager.cc:1905] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:59.697295 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 23086
I20260929 02:55:59.697530 23550 ts_tablet_manager.cc:1905] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:59.698396 23548 ts_tablet_manager.cc:1905] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:55:59.703735 23549 ts_tablet_manager.cc:1918] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:59.703871 23549 log.cc:1196] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/c9fe6d83e8c74ca9b15c45834bdc6a6d
I20260929 02:55:59.704236 23549 ts_tablet_manager.cc:1939] T c9fe6d83e8c74ca9b15c45834bdc6a6d P ec2cf3e1cb5841708d8a88efb52eacd3: Deleting consensus metadata
I20260929 02:55:59.704561 23550 ts_tablet_manager.cc:1918] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:59.704713 23550 log.cc:1196] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/c9fe6d83e8c74ca9b15c45834bdc6a6d
I20260929 02:55:59.704761 23548 ts_tablet_manager.cc:1918] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:55:59.704891 23548 log.cc:1196] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/c9fe6d83e8c74ca9b15c45834bdc6a6d
I20260929 02:55:59.705276 23548 ts_tablet_manager.cc:1939] T c9fe6d83e8c74ca9b15c45834bdc6a6d P fdef5fa80ed941cc8fd4932698bef4e5: Deleting consensus metadata
I20260929 02:55:59.705415 23001 catalog_manager.cc:4841] TS ec2cf3e1cb5841708d8a88efb52eacd3 (127.15.18.66:36045): tablet c9fe6d83e8c74ca9b15c45834bdc6a6d (table test-table [id=6258bb881956487d99b7f9a94e92e2a9]) successfully deleted
I20260929 02:55:59.705711 23550 ts_tablet_manager.cc:1939] T c9fe6d83e8c74ca9b15c45834bdc6a6d P efb1b689a4b54036a4ee37dfe35c1f7a: Deleting consensus metadata
I20260929 02:55:59.706301 23001 catalog_manager.cc:4841] TS fdef5fa80ed941cc8fd4932698bef4e5 (127.15.18.65:36383): tablet c9fe6d83e8c74ca9b15c45834bdc6a6d (table test-table [id=6258bb881956487d99b7f9a94e92e2a9]) successfully deleted
I20260929 02:55:59.707219 23000 catalog_manager.cc:4841] TS efb1b689a4b54036a4ee37dfe35c1f7a (127.15.18.67:34653): tablet c9fe6d83e8c74ca9b15c45834bdc6a6d (table test-table [id=6258bb881956487d99b7f9a94e92e2a9]) successfully deleted
Sep 29 02:55:59 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.65@KRBTEST.COM
I20260929 02:55:59.721431 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:59.699023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.15.18.65:36383)
0929 02:55:59.699207 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:59.699214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:59.707311 (+ 8097us) server_negotiation.cc:407] Connection header received
0929 02:55:59.707497 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:59.707502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:59.707617 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:59.707710 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:59.708959 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.709849 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.711088 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.711492 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.711661 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.715945 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:59.715960 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:59.715976 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:59.716015 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:59.718744 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:59.719410 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:59.719417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:59.719420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:59.719585 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:59.719935 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:59.719940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:59.719944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:59.720328 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:59.720598 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:59.720914 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 02:55:59.721121 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:55:59.722038 23216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:59.844954 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 23086
I20260929 02:55:59.862960 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 23225
Sep 29 02:55:59 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.66@KRBTEST.COM
I20260929 02:55:59.877964 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:59.863710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.15.18.66:36045)
0929 02:55:59.864174 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:59.864178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:59.864194 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:55:59.864246 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:59.864250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:59.864314 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:59.864400 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:59.865747 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.866489 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.868022 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.868492 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.868614 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.872486 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:59.872499 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:59.872515 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:59.872546 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:59.875308 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:59.875997 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:59.876004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:59.876007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:59.876132 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:55:59.876471 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:55:59.876474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:55:59.876476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:55:59.876833 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:55:59.877090 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 02:55:59.877418 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 02:55:59.877631 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260929 02:55:59.878556 23355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:55:59.973346 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 23225
I20260929 02:55:59.987169 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 23364
Sep 29 02:55:59 dist-test-slave-q5f0 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.18.67@KRBTEST.COM
I20260929 02:56:00.002283 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:55:59.987999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.15.18.67:34653)
0929 02:55:59.988172 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:55:59.988176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:55:59.988194 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:55:59.988248 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:55:59.988252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:55:59.988324 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:55:59.988408 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:55:59.989629 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.990382 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.991710 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:55:59.992102 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:55:59.992218 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:55:59.996259 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:55:59.996275 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:55:59.996292 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:55:59.996332 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:55:59.999121 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:00.000189 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:00.000199 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:00.000201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:00.000355 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:00.000810 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:00.000814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:00.000817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:00.001211 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:00.001479 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:00.001806 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 02:56:00.002005 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:56:00.002938 23494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:00.147181 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 23364
I20260929 02:56:00.166823 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 22983
I20260929 02:56:00.167958 23055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:00.282315 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 22983
2026-09-29T02:56:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6300 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:56:00 dist-test-slave-q5f0 krb5kdc[23629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:56:00 dist-test-slave-q5f0 krb5kdc[23629](info): set up 2 sockets
Sep 29 02:56:00 dist-test-slave-q5f0 krb5kdc[23629](info): commencing operation
krb5kdc: starting...
W20260929 02:56:01.288730 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:56:01 dist-test-slave-q5f0 krb5kdc[23629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:56:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:56:01.499424 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:42279
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:56:01.813064 23642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:01.813367 23642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:01.813454 23642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:01.820686 23642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:56:01.820799 23642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:01.820842 23642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:01.820895 23642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:56:01.820941 23642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:56:01.830734 23642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:42279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:01.831984 23642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:01.833261 23642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:01.845101 23647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:01.845029 23650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:01.845069 23648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:01.845865 23642 server_base.cc:1034] running on GCE node
I20260929 02:56:01.846669 23642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:01.848248 23642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:01.849480 23642 hybrid_clock.cc:648] HybridClock initialized: now 1790650561849437 us; error 32 us; skew 500 ppm
I20260929 02:56:01.849879 23642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T02:56:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1637 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:56:02 dist-test-slave-q5f0 krb5kdc[23656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:56:02 dist-test-slave-q5f0 krb5kdc[23656](info): set up 2 sockets
Sep 29 02:56:02 dist-test-slave-q5f0 krb5kdc[23656](info): commencing operation
krb5kdc: starting...
W20260929 02:56:02.924484 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:56:03 dist-test-slave-q5f0 krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:56:03.135630 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:45171
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:45171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:56:03.462232 23669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:03.462529 23669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:03.462620 23669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:03.469877 23669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:56:03.470000 23669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:03.470044 23669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:03.470100 23669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:56:03.470144 23669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:56:03.479981 23669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:45171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:45171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:03.481256 23669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:03.482510 23669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:03.493799 23677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:03.493924 23674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:03.493898 23675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:03.494474 23669 server_base.cc:1034] running on GCE node
I20260929 02:56:03.495182 23669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:03.496628 23669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:03.497834 23669 hybrid_clock.cc:648] HybridClock initialized: now 1790650563497791 us; error 38 us; skew 500 ppm
I20260929 02:56:03.498162 23669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:03 dist-test-slave-q5f0 krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650563, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:03.501219 23669 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:56:03.502818 23669 webserver.cc:460] Webserver started at http://127.15.18.126:35605/ using document root <none> and password file <none>
I20260929 02:56:03.503564 23669 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:03.503686 23669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:03.503976 23669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:03.506940 23669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f"
format_stamp: "Formatted at 2026-09-29 02:56:03 on dist-test-slave-q5f0"
server_key: "454925185148bfc4e85847d2e61dc4f7"
server_key_iv: "b323f43f74117a3992d2cbdd547d1c8d"
server_key_version: "encryptionkey@0"
I20260929 02:56:03.507726 23669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f"
format_stamp: "Formatted at 2026-09-29 02:56:03 on dist-test-slave-q5f0"
server_key: "454925185148bfc4e85847d2e61dc4f7"
server_key_iv: "b323f43f74117a3992d2cbdd547d1c8d"
server_key_version: "encryptionkey@0"
I20260929 02:56:03.512940 23669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:56:03.516270 23684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:03.517625 23669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:56:03.517786 23669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f"
format_stamp: "Formatted at 2026-09-29 02:56:03 on dist-test-slave-q5f0"
server_key: "454925185148bfc4e85847d2e61dc4f7"
server_key_iv: "b323f43f74117a3992d2cbdd547d1c8d"
server_key_version: "encryptionkey@0"
I20260929 02:56:03.517972 23669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:03.548367 23669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:03.552647 23669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:03.552908 23669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:03.569734 23669 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:45171
I20260929 02:56:03.569767 23735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:45171 every 8 connection(s)
I20260929 02:56:03.571049 23669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:56:03.574750 23736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:56:03.576331 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 23669
I20260929 02:56:03.576521 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:56:03.577158 15433 external_mini_cluster.cc:1428] Setting key 6f630f327b6295eec2726df8cc37eedd
I20260929 02:56:03.586566 23736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f: Bootstrap starting.
I20260929 02:56:03.590461 23736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f: Neither blocks nor log segments found. Creating new log.
Sep 29 02:56:03 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:56:03.591722 23736 log.cc:824] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:56:03.594892 23736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f: No bootstrap required, opened a new log
I20260929 02:56:03.600769 23739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:03.579067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.15.18.126:45171)
0929 02:56:03.579881 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:03.579893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:03.579937 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:56:03.581616 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:03.581640 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:03.582053 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:03.582380 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:03.584190 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:03.585513 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:03.587233 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:03.587998 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:03.588155 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:03.592914 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:03.592926 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:03.592947 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:03.593007 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:03.596331 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:03.597168 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:03.597175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:03.597182 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:03.597377 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:03.597777 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:03.597781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:03.597783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:03.598420 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:03.598703 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:03.599180 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 02:56:03.599634 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":371,"threads_started":1}
I20260929 02:56:03.603883 23736 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } }
I20260929 02:56:03.604220 23736 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:56:03.604321 23736 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b4d9c9b5f4446bbfdfc25182f38d8f, State: Initialized, Role: FOLLOWER
I20260929 02:56:03.604893 23736 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } }
I20260929 02:56:03.605110 23736 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:56:03.605232 23736 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:56:03.605374 23736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:56:03.607605 23736 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } }
I20260929 02:56:03.608120 23736 leader_election.cc:302] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9b4d9c9b5f4446bbfdfc25182f38d8f; no voters:
I20260929 02:56:03.608675 23736 leader_election.cc:288] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:56:03.608829 23741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:56:03.609540 23741 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 1 LEADER]: Becoming Leader. State: Replica: d9b4d9c9b5f4446bbfdfc25182f38d8f, State: Running, Role: LEADER
I20260929 02:56:03.610101 23741 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } }
I20260929 02:56:03.610668 23736 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:56:03.612576 23743 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b4d9c9b5f4446bbfdfc25182f38d8f. Latest consensus state: current_term: 1 leader_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } } }
I20260929 02:56:03.612735 23742 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } } }
I20260929 02:56:03.613001 23743 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [sys.catalog]: This master's current role is: LEADER
I20260929 02:56:03.613387 23742 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [sys.catalog]: This master's current role is: LEADER
I20260929 02:56:03.614590 23746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:56:03.619659 23746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:56:03.628692 23746 catalog_manager.cc:1347] Generated new cluster ID: 9d0060726d4847ea98c6b1a0d1d26ce7
I20260929 02:56:03.628845 23746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:56:03.647311 23746 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:56:03.648694 23746 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:56:03.662590 23746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f: Generated new TSK 0
I20260929 02:56:03.663496 23746 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:56:03.727099 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45171
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:56:04.069357 23762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:04.069674 23762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:04.069769 23762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:04.077252 23762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:04.077379 23762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:04.077494 23762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:56:04.087318 23762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:04.088611 23762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:04.090004 23762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:04.103150 23767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:04.103205 23768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:04.103622 23770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:04.104130 23762 server_base.cc:1034] running on GCE node
I20260929 02:56:04.104815 23762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:04.105830 23762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:04.107081 23762 hybrid_clock.cc:648] HybridClock initialized: now 1790650564107014 us; error 61 us; skew 500 ppm
I20260929 02:56:04.107403 23762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:04 dist-test-slave-q5f0 krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650564, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:04.110680 23762 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:56:04.112188 23762 webserver.cc:460] Webserver started at http://127.15.18.65:39847/ using document root <none> and password file <none>
I20260929 02:56:04.112869 23762 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:04.112967 23762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:04.113262 23762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:04.116384 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "79ae6e958ad04c2b8d2858d904c60ccd"
format_stamp: "Formatted at 2026-09-29 02:56:04 on dist-test-slave-q5f0"
server_key: "b15c018437be4d3d93776e6a6acd54da"
server_key_iv: "9bb880de7c213ece44f2f6d010300436"
server_key_version: "encryptionkey@0"
I20260929 02:56:04.117257 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "79ae6e958ad04c2b8d2858d904c60ccd"
format_stamp: "Formatted at 2026-09-29 02:56:04 on dist-test-slave-q5f0"
server_key: "b15c018437be4d3d93776e6a6acd54da"
server_key_iv: "9bb880de7c213ece44f2f6d010300436"
server_key_version: "encryptionkey@0"
I20260929 02:56:04.122502 23762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:56:04.126169 23777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:04.127707 23762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:56:04.127876 23762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "79ae6e958ad04c2b8d2858d904c60ccd"
format_stamp: "Formatted at 2026-09-29 02:56:04 on dist-test-slave-q5f0"
server_key: "b15c018437be4d3d93776e6a6acd54da"
server_key_iv: "9bb880de7c213ece44f2f6d010300436"
server_key_version: "encryptionkey@0"
I20260929 02:56:04.128068 23762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:04.142688 23762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:04.147123 23762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:04.147423 23762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:04.148530 23762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:04.150068 23762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:04.150152 23762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:04.150274 23762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:04.150331 23762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:04.180641 23762 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:34823
I20260929 02:56:04.180668 23889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:34823 every 8 connection(s)
I20260929 02:56:04.182015 23762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:56:04.190989 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 23762
I20260929 02:56:04.191165 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:56:04.191821 15433 external_mini_cluster.cc:1428] Setting key 9b762bae1d946717b95d444040e77ef0
Sep 29 02:56:04 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650564, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:56:04.201220 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:04.184539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:34429 (local address 127.15.18.126:45171)
0929 02:56:04.184947 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:04.184951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:04.185857 (+ 906us) server_negotiation.cc:407] Connection header received
0929 02:56:04.187653 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:04.187659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:04.187744 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:04.187839 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:04.189365 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:04.190048 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:04.191644 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:04.192073 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:04.192184 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:04.195502 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:04.195532 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:04.195537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:04.195574 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:04.198181 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:04.198907 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:04.198913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:04.198916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:04.199033 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:04.199450 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:04.199455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:04.199459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:04.199770 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:04.199900 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:04.200638 (+ 738us) server_negotiation.cc:299] Negotiation successful
0929 02:56:04.200864 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":202,"threads_started":1}
I20260929 02:56:04.202245 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:04.184968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:34429)
0929 02:56:04.185681 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:04.185725 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:04.187371 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:04.188088 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:04.188096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:04.188608 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:04.189180 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:04.189197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:04.190197 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:04.190200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:04.191490 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:04.191502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:04.192293 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:04.192300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:04.192532 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:04.193163 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:04.193192 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:04.195296 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:04.198339 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:04.198349 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:04.198368 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:04.198791 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:04.199158 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:04.199161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:04.199164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:04.199346 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:04.199890 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:04.199899 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:04.200259 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 02:56:04.200502 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 02:56:04.201066 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":281,"threads_started":1}
I20260929 02:56:04.203887 23890 heartbeater.cc:347] Connected to a master server at 127.15.18.126:45171
I20260929 02:56:04.204222 23890 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:04.205047 23890 heartbeater.cc:511] Master 127.15.18.126:45171 requested a full tablet report, sending...
I20260929 02:56:04.207373 23701 ts_manager.cc:194] Registered new tserver with Master: 79ae6e958ad04c2b8d2858d904c60ccd (127.15.18.65:34823)
I20260929 02:56:04.209095 23701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:34429
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:56:04.239168 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45171
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:56:04.558117 23896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:04.558400 23896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:04.558526 23896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:04.565816 23896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:04.565941 23896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:04.566080 23896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:56:04.575946 23896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:04.577234 23896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:04.578549 23896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:04.591786 23902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:04.591869 23904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:04.591890 23901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:04.592895 23896 server_base.cc:1034] running on GCE node
I20260929 02:56:04.593554 23896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:04.594570 23896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:04.595816 23896 hybrid_clock.cc:648] HybridClock initialized: now 1790650564595758 us; error 51 us; skew 500 ppm
I20260929 02:56:04.596194 23896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:04 dist-test-slave-q5f0 krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650564, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:04.599741 23896 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:56:04.601333 23896 webserver.cc:460] Webserver started at http://127.15.18.66:40317/ using document root <none> and password file <none>
I20260929 02:56:04.602051 23896 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:04.602181 23896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:04.602475 23896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:04.605587 23896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "8d5d136a37654dafa8ae25b5f01ba254"
format_stamp: "Formatted at 2026-09-29 02:56:04 on dist-test-slave-q5f0"
server_key: "b59c7ec0287f953ef10407c7e70323e3"
server_key_iv: "88fade40eb0ccd71d491f852908846e2"
server_key_version: "encryptionkey@0"
I20260929 02:56:04.606387 23896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "8d5d136a37654dafa8ae25b5f01ba254"
format_stamp: "Formatted at 2026-09-29 02:56:04 on dist-test-slave-q5f0"
server_key: "b59c7ec0287f953ef10407c7e70323e3"
server_key_iv: "88fade40eb0ccd71d491f852908846e2"
server_key_version: "encryptionkey@0"
I20260929 02:56:04.611833 23896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:56:04.615370 23911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:04.616900 23896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:56:04.617061 23896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "8d5d136a37654dafa8ae25b5f01ba254"
format_stamp: "Formatted at 2026-09-29 02:56:04 on dist-test-slave-q5f0"
server_key: "b59c7ec0287f953ef10407c7e70323e3"
server_key_iv: "88fade40eb0ccd71d491f852908846e2"
server_key_version: "encryptionkey@0"
I20260929 02:56:04.617254 23896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:04.633957 23896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:04.638626 23896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:04.638903 23896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:04.639938 23896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:04.641311 23896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:04.641381 23896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:04.641462 23896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:04.641505 23896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:04.671056 23896 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:40611
I20260929 02:56:04.671093 24023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:40611 every 8 connection(s)
I20260929 02:56:04.672319 23896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:56:04.680011 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 23896
I20260929 02:56:04.680186 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:56:04.680608 15433 external_mini_cluster.cc:1428] Setting key 9fb654ea0255bf14db2e2dedcd2909c9
Sep 29 02:56:04 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650564, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:56:04.692837 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:04.674800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:47669 (local address 127.15.18.126:45171)
0929 02:56:04.674979 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:04.674983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:04.676219 (+ 1236us) server_negotiation.cc:407] Connection header received
0929 02:56:04.678085 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:04.678091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:04.678182 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:04.678273 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:04.680060 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:04.680860 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:04.682833 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:04.683282 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:04.683385 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:04.687436 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:04.687441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:04.687444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:04.687476 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:04.689877 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:04.690582 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:04.690585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:04.690588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:04.690731 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:04.691139 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:04.691142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:04.691144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:04.691408 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:04.691564 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:04.692256 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 02:56:04.692482 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:56:04.693883 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:04.675266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:47669)
0929 02:56:04.676034 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:04.676080 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:04.677777 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:04.678544 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:04.678554 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:04.679090 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:04.679826 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:04.679855 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:04.681058 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:04.681064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:04.682667 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:04.682682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:04.683539 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:04.683546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:04.683837 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:04.684651 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:04.684681 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:04.687211 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:04.690010 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:04.690019 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:04.690038 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:04.690459 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:04.690832 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:04.690837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:04.690841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:04.691028 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:04.691550 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:04.691558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:04.691902 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 02:56:04.692131 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 02:56:04.692685 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":319,"threads_started":1}
I20260929 02:56:04.695415 24024 heartbeater.cc:347] Connected to a master server at 127.15.18.126:45171
I20260929 02:56:04.695761 24024 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:04.696466 24024 heartbeater.cc:511] Master 127.15.18.126:45171 requested a full tablet report, sending...
I20260929 02:56:04.697957 23701 ts_manager.cc:194] Registered new tserver with Master: 8d5d136a37654dafa8ae25b5f01ba254 (127.15.18.66:40611)
I20260929 02:56:04.699070 23701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:47669
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:56:04.727079 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45171
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:56:05.047439 24029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:05.047793 24029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:05.047891 24029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:05.055274 24029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:05.055413 24029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:05.055583 24029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:56:05.065513 24029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:05.066841 24029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:05.068194 24029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:05.081079 24034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:05.081048 24035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:05.081116 24037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:05.082466 24029 server_base.cc:1034] running on GCE node
I20260929 02:56:05.083106 24029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:05.084143 24029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:05.085376 24029 hybrid_clock.cc:648] HybridClock initialized: now 1790650565085320 us; error 51 us; skew 500 ppm
I20260929 02:56:05.085743 24029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:05 dist-test-slave-q5f0 krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650565, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:05.088997 24029 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:56:05.090447 24029 webserver.cc:460] Webserver started at http://127.15.18.67:42499/ using document root <none> and password file <none>
I20260929 02:56:05.091125 24029 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:05.091243 24029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:05.091562 24029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:05.094405 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "0c03ac6ba4744a3e96d884c5e7be8ab6"
format_stamp: "Formatted at 2026-09-29 02:56:05 on dist-test-slave-q5f0"
server_key: "da89b58430a357b2ccbeec112e769c85"
server_key_iv: "dc582482fe1efbedf13fbe9d24b8dac9"
server_key_version: "encryptionkey@0"
I20260929 02:56:05.095146 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "0c03ac6ba4744a3e96d884c5e7be8ab6"
format_stamp: "Formatted at 2026-09-29 02:56:05 on dist-test-slave-q5f0"
server_key: "da89b58430a357b2ccbeec112e769c85"
server_key_iv: "dc582482fe1efbedf13fbe9d24b8dac9"
server_key_version: "encryptionkey@0"
I20260929 02:56:05.100459 24029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:56:05.104005 24044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:05.105401 24029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:56:05.105558 24029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "0c03ac6ba4744a3e96d884c5e7be8ab6"
format_stamp: "Formatted at 2026-09-29 02:56:05 on dist-test-slave-q5f0"
server_key: "da89b58430a357b2ccbeec112e769c85"
server_key_iv: "dc582482fe1efbedf13fbe9d24b8dac9"
server_key_version: "encryptionkey@0"
I20260929 02:56:05.105752 24029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:05.117946 24029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:05.122295 24029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:05.122570 24029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:05.123605 24029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:05.124982 24029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:05.125056 24029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:05.125155 24029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:05.125212 24029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:05.155524 24029 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:39271
I20260929 02:56:05.155575 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:39271 every 8 connection(s)
I20260929 02:56:05.156750 24029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:56:05.157435 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 24029
I20260929 02:56:05.157596 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:56:05.158139 15433 external_mini_cluster.cc:1428] Setting key f0a39fae1a897d98e694c63b045cb6af
Sep 29 02:56:05 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650565, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:56:05.176616 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:05.160018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:42239 (local address 127.15.18.126:45171)
0929 02:56:05.160190 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:05.160196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:05.161431 (+ 1235us) server_negotiation.cc:407] Connection header received
0929 02:56:05.163201 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:05.163206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:05.163288 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:05.163373 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:05.164898 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:05.165621 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:05.167134 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:05.167559 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:05.167670 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:05.171125 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:05.171132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:05.171138 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:05.171178 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:05.173910 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:05.174557 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:05.174560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:05.174562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:05.174681 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:05.175063 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:05.175067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:05.175069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:05.175340 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:05.175459 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:05.176187 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 02:56:05.176362 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:56:05.177498 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:05.160481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:42239)
0929 02:56:05.161274 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:05.161331 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:05.162945 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:05.163657 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:05.163666 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:05.164166 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:05.164737 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:05.164756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:05.165771 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:05.165775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:05.166996 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:05.167012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:05.167766 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:05.167773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:05.168004 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:05.168606 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:05.168627 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:05.170895 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:05.174039 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:05.174046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:05.174059 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:05.174469 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:05.174764 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:05.174767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:05.174769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:05.174941 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:05.175441 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:05.175448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:05.175843 (+ 395us) client_negotiation.cc:769] Sending connection context
0929 02:56:05.176071 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 02:56:05.176456 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":290,"threads_started":1}
I20260929 02:56:05.178838 24157 heartbeater.cc:347] Connected to a master server at 127.15.18.126:45171
I20260929 02:56:05.179113 24157 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:05.179834 24157 heartbeater.cc:511] Master 127.15.18.126:45171 requested a full tablet report, sending...
I20260929 02:56:05.181413 23701 ts_manager.cc:194] Registered new tserver with Master: 0c03ac6ba4744a3e96d884c5e7be8ab6 (127.15.18.67:39271)
I20260929 02:56:05.182452 23701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:42239
I20260929 02:56:05.185539 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.65@KRBTEST.COM" created.
I20260929 02:56:05.212519 23890 heartbeater.cc:503] Master 127.15.18.126:45171 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab.
Entry for principal oryx/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab.
WARNING: no policy specified for oryx/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab.
Entry for principal oryx/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab.
WARNING: no policy specified for oryx/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab.
Entry for principal oryx/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab.
I20260929 02:56:05.318194 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 23762
Sep 29 02:56:05 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:56:05.335330 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:05.319252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.15.18.65:34823)
0929 02:56:05.320019 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:05.320027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:05.320054 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:56:05.320113 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:05.320120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:05.320325 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:05.320457 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:05.322095 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:05.323269 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:05.324633 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:05.325209 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:05.325313 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:05.329285 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:05.329296 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:05.329310 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:05.329360 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:05.332300 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:05.333191 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:05.333199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:05.333202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:05.333331 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:05.333799 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:05.333803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:05.333805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:05.334157 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:05.334394 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:05.334743 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 02:56:05.334982 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":470,"threads_started":1}
I20260929 02:56:05.336553 23885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:05.418424 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 23762
I20260929 02:56:05.434044 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 23896
Sep 29 02:56:05 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:56:05.450340 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:05.434929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.15.18.66:40611)
0929 02:56:05.435438 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:05.435447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:05.435479 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:56:05.435574 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:05.435583 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:05.435869 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:05.436038 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:05.438109 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:05.439160 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:05.440465 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:05.441071 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:05.441198 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:05.445059 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:05.445072 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:05.445086 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:05.445120 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:05.447720 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:05.448328 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:05.448335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:05.448337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:05.448465 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:05.448808 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:05.448812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:05.448814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:05.449141 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:05.449449 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:05.449747 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 02:56:05.450009 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260929 02:56:05.451567 24019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:05.534183 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 23896
I20260929 02:56:05.546649 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24029
Sep 29 02:56:05 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:56:05.563333 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:05.547708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.15.18.67:39271)
0929 02:56:05.548353 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:05.548360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:05.548388 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:56:05.548445 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:05.548451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:05.548670 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:05.548824 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:05.550155 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:05.551211 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:05.552600 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:05.553202 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:05.553332 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:05.557720 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:05.557733 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:05.557749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:05.557795 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:05.560692 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:05.561334 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:05.561341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:05.561343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:05.561472 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:05.561832 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:05.561835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:05.561837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:05.562177 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:05.562469 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:05.562770 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 02:56:05.563021 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":317,"threads_started":1}
I20260929 02:56:05.564634 24152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:05.646323 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 24029
I20260929 02:56:05.659137 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 23669
I20260929 02:56:05.660413 23731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:05.761344 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 23669
I20260929 02:56:05.778263 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:45171
--webserver_interface=127.15.18.126
--webserver_port=35605
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:45171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:56:06.090574 24174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:06.090916 24174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:06.091053 24174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:06.098373 24174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:56:06.098503 24174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:06.098573 24174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:06.098652 24174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:56:06.098708 24174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:56:06.108897 24174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:45171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:45171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=35605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:06.110226 24174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:06.111479 24174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:06.122659 24179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:06.122612 24182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:06.122735 24180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:06.123389 24174 server_base.cc:1034] running on GCE node
I20260929 02:56:06.124191 24174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:06.125643 24174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:06.126856 24174 hybrid_clock.cc:648] HybridClock initialized: now 1790650566126801 us; error 51 us; skew 500 ppm
I20260929 02:56:06.127198 24174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:06 dist-test-slave-q5f0 krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650566, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:06.130183 24174 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:56:06.131592 24174 webserver.cc:460] Webserver started at http://127.15.18.126:35605/ using document root <none> and password file <none>
I20260929 02:56:06.132247 24174 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:06.132354 24174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:06.137643 24174 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 02:56:06.140300 24189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:06.141757 24174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:56:06.141909 24174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f"
format_stamp: "Formatted at 2026-09-29 02:56:03 on dist-test-slave-q5f0"
server_key: "454925185148bfc4e85847d2e61dc4f7"
server_key_iv: "b323f43f74117a3992d2cbdd547d1c8d"
server_key_version: "encryptionkey@0"
I20260929 02:56:06.142836 24174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:06.154633 24174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:06.158838 24174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:06.159082 24174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:06.175787 24174 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:45171
I20260929 02:56:06.175804 24240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:45171 every 8 connection(s)
I20260929 02:56:06.177210 24174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:56:06.178332 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 24174
I20260929 02:56:06.179271 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:34823
--local_ip_for_outbound_sockets=127.15.18.65
--tserver_master_addrs=127.15.18.126:45171
--webserver_port=39847
--webserver_interface=127.15.18.65
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.18.65
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:56:06.185949 24241 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:56:06.189438 24241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f: Bootstrap starting.
I20260929 02:56:06.207852 24241 log.cc:824] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:56:06.216782 24241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:56:06.217334 24241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f: Bootstrap complete.
I20260929 02:56:06.226023 24241 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } }
I20260929 02:56:06.226835 24241 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b4d9c9b5f4446bbfdfc25182f38d8f, State: Initialized, Role: FOLLOWER
I20260929 02:56:06.227456 24241 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } }
I20260929 02:56:06.227700 24241 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:56:06.227823 24241 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:56:06.228087 24241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:56:06.232033 24241 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } }
I20260929 02:56:06.232533 24241 leader_election.cc:302] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9b4d9c9b5f4446bbfdfc25182f38d8f; no voters:
I20260929 02:56:06.233116 24241 leader_election.cc:288] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:56:06.233271 24245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:56:06.235232 24245 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [term 2 LEADER]: Becoming Leader. State: Replica: d9b4d9c9b5f4446bbfdfc25182f38d8f, State: Running, Role: LEADER
I20260929 02:56:06.235383 24241 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:56:06.235812 24245 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } }
I20260929 02:56:06.238381 24246 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } } }
I20260929 02:56:06.238749 24246 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [sys.catalog]: This master's current role is: LEADER
I20260929 02:56:06.238415 24247 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b4d9c9b5f4446bbfdfc25182f38d8f. Latest consensus state: current_term: 2 leader_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b4d9c9b5f4446bbfdfc25182f38d8f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45171 } } }
I20260929 02:56:06.238858 24247 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f [sys.catalog]: This master's current role is: LEADER
I20260929 02:56:06.251262 24253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:56:06.258138 24253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:56:06.259856 24253 catalog_manager.cc:1259] Loaded cluster ID: 9d0060726d4847ea98c6b1a0d1d26ce7
I20260929 02:56:06.260049 24253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:56:06.264430 24253 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:56:06.266070 24253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d9b4d9c9b5f4446bbfdfc25182f38d8f: Loaded TSK: 0
I20260929 02:56:06.267887 24253 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:56:06.523312 24243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:06.523638 24243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:06.523775 24243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:06.531162 24243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:06.531309 24243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:06.531471 24243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:56:06.542599 24243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.65.keytab
--principal=oryx/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:34823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=39847
--tserver_master_addrs=127.15.18.126:45171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:06.543975 24243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:06.545277 24243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:06.560003 24268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:06.560019 24267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:06.561009 24270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:06.561388 24243 server_base.cc:1034] running on GCE node
I20260929 02:56:06.562222 24243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:06.563592 24243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:06.564857 24243 hybrid_clock.cc:648] HybridClock initialized: now 1790650566564797 us; error 54 us; skew 500 ppm
I20260929 02:56:06.565272 24243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:06 dist-test-slave-q5f0 krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650566, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:06.569121 24243 init.cc:377] Logged in from keytab as oryx/127.15.18.65@KRBTEST.COM (short username oryx)
I20260929 02:56:06.572687 24243 webserver.cc:460] Webserver started at http://127.15.18.65:39847/ using document root <none> and password file <none>
I20260929 02:56:06.573408 24243 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:06.573535 24243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:06.579236 24243 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:56:06.582123 24277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:06.583559 24243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:56:06.583735 24243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "79ae6e958ad04c2b8d2858d904c60ccd"
format_stamp: "Formatted at 2026-09-29 02:56:04 on dist-test-slave-q5f0"
server_key: "b15c018437be4d3d93776e6a6acd54da"
server_key_iv: "9bb880de7c213ece44f2f6d010300436"
server_key_version: "encryptionkey@0"
I20260929 02:56:06.584641 24243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:06.599653 24243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:06.603873 24243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:06.604132 24243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:06.605108 24243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:06.606519 24243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:06.606613 24243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:06.606732 24243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:06.606799 24243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:06.637068 24243 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:34823
I20260929 02:56:06.637135 24389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:34823 every 8 connection(s)
I20260929 02:56:06.638608 24243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:56:06.642570 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 24243
I20260929 02:56:06.643128 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:40611
--local_ip_for_outbound_sockets=127.15.18.66
--tserver_master_addrs=127.15.18.126:45171
--webserver_port=40317
--webserver_interface=127.15.18.66
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.18.66
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:56:06 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:06 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:56:06.970642 24394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:06.970937 24394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:06.971065 24394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:06.978194 24394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:06.978324 24394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:06.978472 24394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:56:06.988226 24394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.66.keytab
--principal=oryx/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:40611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=40317
--tserver_master_addrs=127.15.18.126:45171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:06.989663 24394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:06.990962 24394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:07.005263 24400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:07.005626 24401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:07.006691 24394 server_base.cc:1034] running on GCE node
W20260929 02:56:07.007740 24403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:07.008718 24394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:07.010161 24394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:07.011420 24394 hybrid_clock.cc:648] HybridClock initialized: now 1790650567011357 us; error 44 us; skew 500 ppm
I20260929 02:56:07.011889 24394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:07 dist-test-slave-q5f0 krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650567, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:07.020891 24394 init.cc:377] Logged in from keytab as oryx/127.15.18.66@KRBTEST.COM (short username oryx)
I20260929 02:56:07.023234 24394 webserver.cc:460] Webserver started at http://127.15.18.66:40317/ using document root <none> and password file <none>
I20260929 02:56:07.024384 24394 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:07.024559 24394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:07.033175 24394 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:56:07.037377 24410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:07.039413 24394 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260929 02:56:07.039673 24394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "8d5d136a37654dafa8ae25b5f01ba254"
format_stamp: "Formatted at 2026-09-29 02:56:04 on dist-test-slave-q5f0"
server_key: "b59c7ec0287f953ef10407c7e70323e3"
server_key_iv: "88fade40eb0ccd71d491f852908846e2"
server_key_version: "encryptionkey@0"
I20260929 02:56:07.041260 24394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:07.074368 24394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:07.081334 24394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:07.081784 24394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:07.083220 24394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:07.085230 24394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:07.085386 24394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:07.085510 24394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:07.085600 24394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:07.120339 24394 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:40611
I20260929 02:56:07.120366 24522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:40611 every 8 connection(s)
I20260929 02:56:07.122601 24394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:56:07.127081 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 24394
I20260929 02:56:07.127849 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:39271
--local_ip_for_outbound_sockets=127.15.18.67
--tserver_master_addrs=127.15.18.126:45171
--webserver_port=42499
--webserver_interface=127.15.18.67
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.18.67
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:56:07 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:07 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:56:07.660918 24393 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.009s user 0.001s sys 0.015s
I20260929 02:56:07.662902 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:06.641511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:55627)
0929 02:56:06.642305 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:06.642350 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:06.644083 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:06.645660 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:06.645668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:06.646241 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:06.646801 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:06.646819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:06.648916 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:06.648921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:06.650165 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:06.650178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:06.651326 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:06.651333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:06.651614 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:06.652285 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:06.652319 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:07.661700 (+1009381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":336,"threads_started":1}
W20260929 02:56:07.662928 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:06.641452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:55627 (local address 127.15.18.126:45171)
0929 02:56:06.642079 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:06.642094 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:06.642472 (+ 378us) server_negotiation.cc:407] Connection header received
0929 02:56:06.644640 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:06.644658 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:06.645101 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:06.645470 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:06.646980 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:06.648742 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:06.650319 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:06.651091 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:06.651263 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:07.662147 (+1010884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:55627: BlockingRecv error: recv got EOF from 127.15.18.65:55627 (error 108)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":240,"threads_started":1}
W20260929 02:56:07.664286 24390 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:45171 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.18.126:45171: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Sep 29 02:56:07 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:07 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:07.673491 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:07.664988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:55679)
0929 02:56:07.665346 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:07.665361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:07.665482 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:07.665795 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:07.665799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:07.666168 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:07.666337 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:07.666346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.667344 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.667349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:07.668334 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:07.668346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.669050 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.669057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:07.669198 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:07.669783 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:07.669801 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:07.673187 (+ 3386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":69}
W20260929 02:56:07.673619 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:07.665077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:55679 (local address 127.15.18.126:45171)
0929 02:56:07.665230 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:07.665237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:07.665435 (+ 198us) server_negotiation.cc:407] Connection header received
0929 02:56:07.665602 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:07.665606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:07.665665 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:07.665744 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:07.666481 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.667198 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.668467 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.668884 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.668989 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:07.673399 (+ 4410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:55679: BlockingRecv error: recv got EOF from 127.15.18.65:55679 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 02:56:07 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:07 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:07.682507 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:07.674345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:52765)
0929 02:56:07.674661 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:07.674678 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:07.674788 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:07.675225 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:07.675228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:07.675493 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:07.675681 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:07.675689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.676656 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.676661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:07.677630 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:07.677643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.678303 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.678310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:07.678479 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:07.679018 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:07.679036 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:07.682229 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223}
W20260929 02:56:07.682693 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:07.674510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:52765 (local address 127.15.18.126:45171)
0929 02:56:07.674660 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:07.674665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:07.674739 (+ 74us) server_negotiation.cc:407] Connection header received
0929 02:56:07.674890 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:07.674896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:07.674998 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:07.675082 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:07.675808 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.676522 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.677765 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.678166 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.678270 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:07.682484 (+ 4214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:52765: BlockingRecv error: recv got EOF from 127.15.18.65:52765 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 02:56:07.683058 24390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:56:07.697101 24527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:07.697707 24527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:07.697912 24527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:07.711714 24527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:07.711913 24527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:07.712139 24527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:56:07.730516 24527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/krb5kdc/oryx_127.15.18.67.keytab
--principal=oryx/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:39271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=42499
--tserver_master_addrs=127.15.18.126:45171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:07.732777 24527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:07.734918 24527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:07.752532 24536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:07.752467 24534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:07.753006 24533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:07.757236 24527 server_base.cc:1034] running on GCE node
I20260929 02:56:07.757887 24527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:07.758960 24527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:07.760324 24527 hybrid_clock.cc:648] HybridClock initialized: now 1790650567760254 us; error 75 us; skew 500 ppm
I20260929 02:56:07.760637 24527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:07 dist-test-slave-q5f0 krb5kdc[23656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650567, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:07.763871 24527 init.cc:377] Logged in from keytab as oryx/127.15.18.67@KRBTEST.COM (short username oryx)
I20260929 02:56:07.765511 24527 webserver.cc:460] Webserver started at http://127.15.18.67:42499/ using document root <none> and password file <none>
I20260929 02:56:07.766225 24527 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:07.766366 24527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:07.772423 24527 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:56:07.775998 24543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:07.777647 24527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:56:07.777813 24527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "0c03ac6ba4744a3e96d884c5e7be8ab6"
format_stamp: "Formatted at 2026-09-29 02:56:05 on dist-test-slave-q5f0"
server_key: "da89b58430a357b2ccbeec112e769c85"
server_key_iv: "dc582482fe1efbedf13fbe9d24b8dac9"
server_key_version: "encryptionkey@0"
I20260929 02:56:07.778766 24527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:07.807329 24527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:07.811842 24527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:07.812108 24527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:07.813284 24527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:07.814709 24527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:07.814803 24527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:07.814908 24527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:07.814978 24527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:07.851409 24527 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:39271
I20260929 02:56:07.851495 24655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:39271 every 8 connection(s)
I20260929 02:56:07.852964 24527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:56:07.858537 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 24527
Sep 29 02:56:07 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:07 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:07.882134 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:07.859564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.15.18.126:45171)
0929 02:56:07.860051 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:07.860057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:07.862069 (+ 2012us) server_negotiation.cc:407] Connection header received
0929 02:56:07.862231 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:07.862238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:07.862331 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:07.862464 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:07.864372 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.865766 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.867458 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.868261 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.868401 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:07.873921 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:07.873934 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:07.873967 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:07.874034 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:07.878536 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:07.879429 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:07.879437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:07.879445 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:07.879672 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:07.880057 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:07.880061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:07.880063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:07.880551 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:07.880849 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:07.881289 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 02:56:07.881818 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
W20260929 02:56:08.254669 24526 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.115s user 0.005s sys 0.011s
W20260929 02:56:08.256165 24525 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:07.125986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:45473 (local address 127.15.18.126:45171)
0929 02:56:07.126420 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:07.126424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:07.130727 (+ 4303us) server_negotiation.cc:407] Connection header received
0929 02:56:07.130809 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:07.130813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:07.130892 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:07.130992 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:07.133793 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.134792 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.136839 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.137612 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.137769 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.255919 (+1118150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:45473: BlockingRecv error: recv got EOF from 127.15.18.66:45473 (error 108)
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":211,"threads_started":1}
I20260929 02:56:08.257105 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:07.126894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:45473)
0929 02:56:07.127953 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:07.128018 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:07.130344 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:07.131415 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:07.131431 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:07.132480 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:07.133513 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:07.133542 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.134977 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.134981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:07.136661 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:07.136678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.137898 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.137906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:07.138189 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:07.139067 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:07.139101 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:08.255753 (+1116652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":446,"threads_started":1}
W20260929 02:56:08.258114 24523 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:45171 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.18.126:45171: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Sep 29 02:56:08 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:08 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:08.269575 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:08.258792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:56797)
0929 02:56:08.259121 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:08.259136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:08.259245 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:08.259700 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:08.259705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:08.260093 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:08.260264 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:08.260274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.261884 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.261888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:08.263107 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:08.263130 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.264655 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.264662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:08.264872 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.265648 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:08.265672 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:08.269347 (+ 3675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:56:08.269953 24525 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:08.258903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:56797 (local address 127.15.18.126:45171)
0929 02:56:08.259043 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:08.259048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:08.259207 (+ 159us) server_negotiation.cc:407] Connection header received
0929 02:56:08.259359 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:08.259365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:08.259460 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:08.259633 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:08.260401 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.261686 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.263358 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.264367 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.264522 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.269679 (+ 5157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:56797: BlockingRecv error: recv got EOF from 127.15.18.66:56797 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 02:56:08 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:08 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:08.278908 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:08.270342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:40803)
0929 02:56:08.270622 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:08.270638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:08.270742 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:08.271084 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:08.271089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:08.271399 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:08.271613 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:08.271623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.272613 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.272618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:08.273556 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:08.273567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.274209 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.274215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:08.274386 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.275001 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:08.275023 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:08.278629 (+ 3606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168}
W20260929 02:56:08.279073 24525 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:08.270655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:40803 (local address 127.15.18.126:45171)
0929 02:56:08.270819 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:08.270824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:08.270840 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:56:08.270889 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:08.270893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:08.270956 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:08.271033 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:08.271750 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.272483 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.273680 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.274079 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.274180 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.278823 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:40803: BlockingRecv error: recv got EOF from 127.15.18.66:40803 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 02:56:08.279633 24523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 02:56:08 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:08 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:56:08.694767 24525 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:08.683861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:48179 (local address 127.15.18.126:45171)
0929 02:56:08.684052 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:08.684058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:08.684623 (+ 565us) server_negotiation.cc:407] Connection header received
0929 02:56:08.684811 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:08.684815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:08.684895 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:08.684976 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:08.686036 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.686772 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.688886 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.689340 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.689480 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.694586 (+ 5106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:48179: BlockingRecv error: recv got EOF from 127.15.18.65:48179 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:56:08.694795 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:08.683703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:48179)
0929 02:56:08.684504 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:08.684529 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:08.684697 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:08.685030 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:08.685036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:08.685654 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:08.685872 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:08.685885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.686894 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.686900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:08.688705 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:08.688725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.689557 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.689565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:08.689726 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.690348 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:08.690370 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:08.694393 (+ 4023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":489,"threads_started":1}
W20260929 02:56:08.884694 24658 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.018s user 0.002s sys 0.010s
W20260929 02:56:08.885779 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:07.855653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:48257 (local address 127.15.18.126:45171)
0929 02:56:07.855860 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:07.855866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:07.857104 (+ 1238us) server_negotiation.cc:407] Connection header received
0929 02:56:07.859351 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:07.859357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:07.859440 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:07.859550 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:07.861559 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.862512 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.864826 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.865289 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.865398 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.885557 (+1020159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:48257: BlockingRecv error: recv got EOF from 127.15.18.67:48257 (error 108)
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":68}
I20260929 02:56:08.886268 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:07.856143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:48257)
0929 02:56:07.856927 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:07.856973 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:07.859032 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:07.860258 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:07.860268 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:07.860777 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:07.861365 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:07.861384 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.862759 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.862765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:07.864679 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:07.864694 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:07.865508 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:07.865515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:07.865750 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:07.866571 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:07.866603 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:08.885386 (+1018783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":314,"threads_started":1}
W20260929 02:56:08.887115 24656 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:45171 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.18.126:45171: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Sep 29 02:56:08 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:08 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:08.896435 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:08.887824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:50833)
0929 02:56:08.887987 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:08.888003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:08.888121 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:08.888469 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:08.888473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:08.888935 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:08.889108 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:08.889120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.890489 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.890493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:08.891442 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:08.891454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.892214 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.892219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:08.892350 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.892915 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:08.892933 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:08.896199 (+ 3266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:56:08.896605 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:08.887972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:50833 (local address 127.15.18.126:45171)
0929 02:56:08.888201 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:08.888206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:08.888224 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:56:08.888276 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:08.888279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:08.888341 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:08.888423 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:08.889260 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.890364 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.891597 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.892018 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.892113 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.896404 (+ 4291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:50833: BlockingRecv error: recv got EOF from 127.15.18.67:50833 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 29 02:56:08 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:08 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:08.905344 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:08.897114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:48275)
0929 02:56:08.897271 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:08.897291 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:08.897382 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:08.897637 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:08.897640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:08.897943 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:08.898095 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:08.898103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.899017 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.899021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:08.900016 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:08.900028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.900687 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.900692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:08.900858 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.901516 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:08.901539 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:08.905124 (+ 3585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:56:08.905534 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:08.897208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:48275 (local address 127.15.18.126:45171)
0929 02:56:08.897356 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:08.897360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:08.897376 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:56:08.897473 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:08.897477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:08.897531 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:08.897603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:08.898202 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.898917 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.900136 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:08.900547 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:08.900638 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:08.905336 (+ 4698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:48275: BlockingRecv error: recv got EOF from 127.15.18.67:48275 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 02:56:08.905673 24656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 02:56:09 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:09 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:09.289426 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:09.280259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:44753)
0929 02:56:09.281168 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:09.281183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:09.281313 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:09.281636 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:09.281641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:09.282129 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:09.282298 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:09.282310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.283330 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.283336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:09.284313 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:09.284324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.285008 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.285012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:09.285124 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:09.285665 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:09.285682 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:09.289156 (+ 3474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":686,"threads_started":1}
W20260929 02:56:09.289552 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:09.280428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:44753 (local address 127.15.18.126:45171)
0929 02:56:09.280668 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:09.280674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:09.281287 (+ 613us) server_negotiation.cc:407] Connection header received
0929 02:56:09.281435 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:09.281440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:09.281513 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:09.281602 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:09.282442 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.283213 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.284442 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.284872 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.284965 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:09.289358 (+ 4393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:44753: BlockingRecv error: recv got EOF from 127.15.18.66:44753 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 02:56:09 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:09 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:09.705302 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:09.695919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:46933)
0929 02:56:09.696537 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:09.696558 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:09.696692 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:09.697046 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:09.697051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:09.697542 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:09.697785 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:09.697798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.699076 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.699080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:09.700082 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:09.700097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.700875 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.700880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:09.701005 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:09.701570 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:09.701586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:09.705043 (+ 3457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":319,"threads_started":1}
W20260929 02:56:09.705435 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:09.696036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:46933 (local address 127.15.18.126:45171)
0929 02:56:09.696283 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:09.696290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:09.696631 (+ 341us) server_negotiation.cc:407] Connection header received
0929 02:56:09.696818 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:09.696822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:09.696888 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:09.696981 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:09.697960 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.698941 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.700235 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.700722 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.700825 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:09.705218 (+ 4393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:46933: BlockingRecv error: recv got EOF from 127.15.18.65:46933 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 29 02:56:09 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:09 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:09.915128 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:09.906302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:45083)
0929 02:56:09.906847 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:09.906867 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:09.907007 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:09.907399 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:09.907404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:09.907913 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:09.908060 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:09.908069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.909067 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.909070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:09.910020 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:09.910032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.910816 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.910821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:09.910957 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:09.911488 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:09.911505 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:09.914787 (+ 3282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":297,"threads_started":1}
W20260929 02:56:09.915220 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:09.906373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:45083 (local address 127.15.18.126:45171)
0929 02:56:09.906562 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:09.906568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:09.906978 (+ 410us) server_negotiation.cc:407] Connection header received
0929 02:56:09.907161 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:09.907167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:09.907253 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:09.907336 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:09.908194 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.908955 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.910170 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:09.910596 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:09.910731 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:09.915004 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:45083: BlockingRecv error: recv got EOF from 127.15.18.67:45083 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 02:56:10 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:10 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:10.299651 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:10.290506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:46461)
0929 02:56:10.291041 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:10.291061 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:10.291220 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:10.291646 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:10.291651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:10.292146 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:10.292374 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:10.292389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.293418 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.293423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:10.294363 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:10.294374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.295080 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.295085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:10.295221 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:10.295927 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:10.295945 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:10.299308 (+ 3363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":295,"threads_started":1}
W20260929 02:56:10.299883 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:10.290743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:46461 (local address 127.15.18.126:45171)
0929 02:56:10.290934 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:10.290940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:10.291173 (+ 233us) server_negotiation.cc:407] Connection header received
0929 02:56:10.291341 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:10.291347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:10.291423 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:10.291555 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:10.292534 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.293282 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.294508 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.294928 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.295020 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:10.299651 (+ 4631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:46461: BlockingRecv error: recv got EOF from 127.15.18.66:46461 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 02:56:10 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:10 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:10.715382 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:10.706543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:44563)
0929 02:56:10.707142 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:10.707163 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:10.707309 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:10.707650 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:10.707656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:10.708038 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:10.708261 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:10.708273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.709340 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.709344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:10.710320 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:10.710335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.711077 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.711082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:10.711236 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:10.711886 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:10.711905 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:10.715143 (+ 3238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":512,"threads_started":1}
W20260929 02:56:10.715581 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:10.706830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:44563 (local address 127.15.18.126:45171)
0929 02:56:10.707057 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:10.707064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:10.707258 (+ 194us) server_negotiation.cc:407] Connection header received
0929 02:56:10.707426 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:10.707430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:10.707491 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:10.707587 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:10.708401 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.709149 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.710475 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.710918 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.711016 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:10.715354 (+ 4338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:44563: BlockingRecv error: recv got EOF from 127.15.18.65:44563 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 02:56:10 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:10 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:10.924595 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:10.916145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:49261)
0929 02:56:10.916613 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:10.916629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:10.916755 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:10.917088 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:10.917092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:10.917503 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:10.917683 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:10.917692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.918661 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.918666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:10.919648 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:10.919661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.920330 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.920334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:10.920446 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:10.920998 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:10.921014 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:10.924361 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":261,"threads_started":1}
W20260929 02:56:10.924767 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:10.916213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:49261 (local address 127.15.18.126:45171)
0929 02:56:10.916411 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:10.916416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:10.916710 (+ 294us) server_negotiation.cc:407] Connection header received
0929 02:56:10.916871 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:10.916875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:10.916933 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:10.917034 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:10.917809 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.918513 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.919778 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:10.920213 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:10.920307 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:10.924550 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:49261: BlockingRecv error: recv got EOF from 127.15.18.67:49261 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 02:56:11 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:11 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:11.309059 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:11.300591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:36679)
0929 02:56:11.301138 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:11.301155 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:11.301267 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:11.301730 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:11.301733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:11.302082 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:11.302239 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:11.302248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.303292 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.303296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:11.304205 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:11.304216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.304856 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.304860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:11.304988 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:11.305545 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:11.305561 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:11.308761 (+ 3200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":317,"threads_started":1}
W20260929 02:56:11.309248 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:11.300707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:36679 (local address 127.15.18.126:45171)
0929 02:56:11.300881 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:11.300885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:11.301233 (+ 348us) server_negotiation.cc:407] Connection header received
0929 02:56:11.301434 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:11.301438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:11.301571 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:11.301709 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:11.302389 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.303148 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.304319 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.304747 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.304835 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:11.308978 (+ 4143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:36679: BlockingRecv error: recv got EOF from 127.15.18.66:36679 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 02:56:11 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:11 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:11.725090 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:11.716443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:49457)
0929 02:56:11.716950 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:11.716966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:11.717088 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:11.717431 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:11.717436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:11.717801 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:11.717963 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:11.717971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.718941 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.718944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:11.719978 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:11.719991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.720641 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.720647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:11.720811 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:11.721412 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:11.721428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:11.724741 (+ 3313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":276,"threads_started":1}
W20260929 02:56:11.725145 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:11.716578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:49457 (local address 127.15.18.126:45171)
0929 02:56:11.716756 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:11.716763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:11.717046 (+ 283us) server_negotiation.cc:407] Connection header received
0929 02:56:11.717194 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:11.717200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:11.717290 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:11.717369 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:11.718100 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.718814 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.720105 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.720508 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.720613 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:11.724956 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:49457: BlockingRecv error: recv got EOF from 127.15.18.65:49457 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 02:56:11 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:11 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:11.934182 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:11.925646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:33403)
0929 02:56:11.926161 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:11.926181 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:11.926329 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:11.926654 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:11.926658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:11.927025 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:11.927200 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:11.927209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.928229 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.928235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:11.929156 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:11.929168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.929821 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.929827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:11.930002 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:11.930537 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:11.930554 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:11.933930 (+ 3376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"spinlock_wait_cycles":30080,"thread_start_us":372,"threads_started":1}
W20260929 02:56:11.934356 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:11.925763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:33403 (local address 127.15.18.126:45171)
0929 02:56:11.925956 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:11.925962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:11.926265 (+ 303us) server_negotiation.cc:407] Connection header received
0929 02:56:11.926456 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:11.926460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:11.926520 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:11.926596 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:11.927345 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.928105 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.929291 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:11.929689 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:11.929793 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:11.934116 (+ 4323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:33403: BlockingRecv error: recv got EOF from 127.15.18.67:33403 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 02:56:12 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:12 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:12.323170 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:12.310212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:40287)
0929 02:56:12.310742 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:12.310765 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:12.310874 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:12.311218 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:12.311222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:12.311638 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:12.311812 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:12.311821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.312886 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.312891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:12.313916 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:12.313928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.314656 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.314661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:12.314789 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:12.319094 (+ 4305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:12.319114 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:12.322880 (+ 3766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":294,"threads_started":1}
W20260929 02:56:12.323348 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:12.310313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:40287 (local address 127.15.18.126:45171)
0929 02:56:12.310481 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:12.310485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:12.310871 (+ 386us) server_negotiation.cc:407] Connection header received
0929 02:56:12.310963 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:12.310967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:12.311033 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:12.311138 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:12.311954 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.312743 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.314062 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.314484 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.314580 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:12.323098 (+ 8518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:40287: BlockingRecv error: recv got EOF from 127.15.18.66:40287 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 02:56:12 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:12 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:12.734591 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:12.726023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:50277)
0929 02:56:12.726511 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:12.726528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:12.726644 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:12.726919 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:12.726923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:12.727352 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:12.727498 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:12.727541 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.728540 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.728544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:12.729489 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:12.729500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.730214 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.730218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:12.730365 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:12.730947 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:12.730970 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:12.734350 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":258,"threads_started":1}
W20260929 02:56:12.734706 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:12.726138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:50277 (local address 127.15.18.126:45171)
0929 02:56:12.726342 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:12.726348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:12.726644 (+ 296us) server_negotiation.cc:407] Connection header received
0929 02:56:12.726729 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:12.726732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:12.726795 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:12.726887 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:12.727686 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.728437 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.729632 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.730083 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.730179 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:12.734526 (+ 4347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:50277: BlockingRecv error: recv got EOF from 127.15.18.65:50277 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 02:56:12 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:12 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:12.944293 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:12.935155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:60011)
0929 02:56:12.935695 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:12.935712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:12.935825 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:12.936110 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:12.936112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:12.936471 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:12.936676 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:12.936690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.937706 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.937711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:12.938778 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:12.938792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.939474 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.939478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:12.939682 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:12.940318 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:12.940341 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:12.943952 (+ 3611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":241,"threads_started":1}
W20260929 02:56:12.944368 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:12.935269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:60011 (local address 127.15.18.126:45171)
0929 02:56:12.935564 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:12.935571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:12.935789 (+ 218us) server_negotiation.cc:407] Connection header received
0929 02:56:12.935941 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:12.935945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:12.936007 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:12.936086 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:12.936800 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.937552 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.938925 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:12.939329 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:12.939426 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:12.944160 (+ 4734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:60011: BlockingRecv error: recv got EOF from 127.15.18.67:60011 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 29 02:56:13 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:13 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:13.333189 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:13.324429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:42523)
0929 02:56:13.325118 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:13.325137 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:13.325281 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:13.325621 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:13.325624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:13.326159 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:13.326310 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:13.326319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.327280 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.327283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:13.328200 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:13.328210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.328906 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.328910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:13.329056 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:13.329606 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:13.329623 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:13.332970 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":432,"threads_started":1}
W20260929 02:56:13.333391 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:13.324694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:42523 (local address 127.15.18.126:45171)
0929 02:56:13.324903 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:13.324909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:13.325264 (+ 355us) server_negotiation.cc:407] Connection header received
0929 02:56:13.325394 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:13.325399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:13.325482 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:13.325566 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:13.326440 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.327177 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.328343 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.328777 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.328873 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:13.333171 (+ 4298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:42523: BlockingRecv error: recv got EOF from 127.15.18.66:42523 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 02:56:13 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:13 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:13.744531 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:13.735644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:59863)
0929 02:56:13.736057 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:13.736081 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:13.736200 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:13.736473 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:13.736477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:13.736898 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:13.737089 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:13.737098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.738124 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.738146 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:13.739087 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:13.739101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.739977 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.739982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:13.740133 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:13.740761 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:13.740783 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:13.744282 (+ 3499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":223,"threads_started":1}
W20260929 02:56:13.744704 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:13.735680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:59863 (local address 127.15.18.126:45171)
0929 02:56:13.735886 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:13.735892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:13.736155 (+ 263us) server_negotiation.cc:407] Connection header received
0929 02:56:13.736285 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:13.736288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:13.736350 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:13.736435 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:13.737258 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.737976 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.739244 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.739803 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.739922 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:13.744511 (+ 4589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:59863: BlockingRecv error: recv got EOF from 127.15.18.65:59863 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 02:56:13 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:13 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:13.954005 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:13.945209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:54861)
0929 02:56:13.945731 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:13.945747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:13.945878 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:13.946232 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:13.946237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:13.946666 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:13.946823 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:13.946832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.948175 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.948181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:13.949261 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:13.949274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.949914 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.949918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:13.950036 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:13.950552 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:13.950569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:13.953777 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":258,"threads_started":1}
W20260929 02:56:13.954160 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:13.945342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:54861 (local address 127.15.18.126:45171)
0929 02:56:13.945557 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:13.945563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:13.945828 (+ 265us) server_negotiation.cc:407] Connection header received
0929 02:56:13.946008 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:13.946013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:13.946075 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:13.946172 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:13.946958 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.948015 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.949392 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:13.949788 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:13.949892 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:13.953947 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:54861: BlockingRecv error: recv got EOF from 127.15.18.67:54861 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 02:56:14 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:14 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:14.343086 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:14.334024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:59165)
0929 02:56:14.334487 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:14.334505 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:14.334638 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:14.335063 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:14.335067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:14.335438 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:14.335647 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:14.335656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.336705 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.336709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:14.337628 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:14.337640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.338381 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.338387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:14.338549 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:14.339185 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:14.339205 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:14.342861 (+ 3656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":243,"threads_started":1}
W20260929 02:56:14.343230 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:14.334175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:59165 (local address 127.15.18.126:45171)
0929 02:56:14.334406 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:14.334412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:14.334582 (+ 170us) server_negotiation.cc:407] Connection header received
0929 02:56:14.334825 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:14.334829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:14.334893 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:14.334978 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:14.335780 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.336569 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.337761 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.338186 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.338328 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:14.343039 (+ 4711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:59165: BlockingRecv error: recv got EOF from 127.15.18.66:59165 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 02:56:14 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:14 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:14.754443 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:14.745438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:37001)
0929 02:56:14.746127 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:14.746143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:14.746295 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:14.746642 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:14.746646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:14.747131 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:14.747304 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:14.747314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.748455 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.748461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:14.749419 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:14.749431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.750119 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.750126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:14.750262 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:14.750807 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:14.750824 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:14.754107 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":336,"threads_started":1}
W20260929 02:56:14.754550 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:14.745523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:37001 (local address 127.15.18.126:45171)
0929 02:56:14.745726 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:14.745730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:14.746264 (+ 534us) server_negotiation.cc:407] Connection header received
0929 02:56:14.746430 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:14.746435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:14.746513 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:14.746605 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:14.747442 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.748323 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.749546 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.749963 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.750068 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:14.754333 (+ 4265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:37001: BlockingRecv error: recv got EOF from 127.15.18.65:37001 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 02:56:14 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:14 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:14.963691 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:14.954956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:39499)
0929 02:56:14.955416 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:14.955432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:14.955588 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:14.955937 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:14.955942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:14.956353 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:14.956520 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:14.956530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.957489 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.957495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:14.958431 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:14.958443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.959120 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.959126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:14.959286 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:14.959859 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:14.959877 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:14.963385 (+ 3508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":246,"threads_started":1}
W20260929 02:56:14.963835 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:14.955083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:39499 (local address 127.15.18.126:45171)
0929 02:56:14.955285 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:14.955291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:14.955529 (+ 238us) server_negotiation.cc:407] Connection header received
0929 02:56:14.955711 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:14.955716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:14.955794 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:14.955876 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:14.956662 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.957352 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.958580 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:14.958964 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:14.959067 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:14.963621 (+ 4554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:39499: BlockingRecv error: recv got EOF from 127.15.18.67:39499 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 02:56:15 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:15 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:15.353595 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:15.344199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:47551)
0929 02:56:15.344787 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:15.344803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:15.344918 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:15.345206 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:15.345209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:15.345703 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:15.345896 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:15.345906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.347227 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.347234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:15.348388 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:15.348402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.349190 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.349195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:15.349334 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:15.349926 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:15.349946 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:15.353356 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":304,"threads_started":1}
W20260929 02:56:15.353822 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:15.344289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:47551 (local address 127.15.18.126:45171)
0929 02:56:15.344510 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:15.344515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:15.344903 (+ 388us) server_negotiation.cc:407] Connection header received
0929 02:56:15.345013 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:15.345016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:15.345079 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:15.345176 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:15.346041 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.347085 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.348530 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.349030 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.349146 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:15.353562 (+ 4416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:47551: BlockingRecv error: recv got EOF from 127.15.18.66:47551 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 02:56:15 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:15 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:15.764024 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:15.755470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:43707)
0929 02:56:15.756013 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:15.756029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:15.756159 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:15.756442 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:15.756446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:15.756831 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:15.756992 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:15.757000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.758026 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.758030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:15.758962 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:15.758974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.759706 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.759711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:15.759829 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:15.760371 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:15.760386 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:15.763759 (+ 3373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":273,"threads_started":1}
W20260929 02:56:15.764358 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:15.755607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:43707 (local address 127.15.18.126:45171)
0929 02:56:15.755803 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:15.755809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:15.756120 (+ 311us) server_negotiation.cc:407] Connection header received
0929 02:56:15.756254 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:15.756258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:15.756319 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:15.756399 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:15.757170 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.757886 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.759089 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.759547 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.759656 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:15.764097 (+ 4441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:43707: BlockingRecv error: recv got EOF from 127.15.18.65:43707 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 02:56:15 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:15 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:15.973414 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:15.964699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:32847)
0929 02:56:15.965264 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:15.965280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:15.965407 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:15.965717 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:15.965722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:15.966130 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:15.966290 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:15.966299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.967299 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.967304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:15.968294 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:15.968306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.968989 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.968995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:15.969145 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:15.969726 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:15.969742 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:15.973111 (+ 3369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":275,"threads_started":1}
W20260929 02:56:15.973559 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:15.964885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:32847 (local address 127.15.18.126:45171)
0929 02:56:15.965116 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:15.965122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:15.965354 (+ 232us) server_negotiation.cc:407] Connection header received
0929 02:56:15.965530 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:15.965535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:15.965596 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:15.965674 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:15.966431 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.967162 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.968419 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:15.968850 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:15.968954 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:15.973343 (+ 4389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:32847: BlockingRecv error: recv got EOF from 127.15.18.67:32847 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 02:56:16 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:16 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:56:16.364161 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:16.354665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:39903 (local address 127.15.18.126:45171)
0929 02:56:16.354877 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:16.354882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:16.355173 (+ 291us) server_negotiation.cc:407] Connection header received
0929 02:56:16.355299 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:16.355303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:16.355368 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:16.355452 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:16.356363 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.357302 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.358614 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.359071 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.359178 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:16.363917 (+ 4739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:39903: BlockingRecv error: recv got EOF from 127.15.18.66:39903 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:56:16.364495 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:16.354581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:39903)
0929 02:56:16.355050 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:16.355067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:16.355197 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:16.355583 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:16.355588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:16.356001 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:16.356184 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:16.356194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.357476 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.357481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:16.358448 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:16.358462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.359275 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.359281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:16.359410 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:16.360086 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:16.360106 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:16.364179 (+ 4073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
Sep 29 02:56:16 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:16 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:16.774047 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:16.765107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:43467)
0929 02:56:16.765587 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:16.765605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:16.765723 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:16.766079 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:16.766083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:16.766466 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:16.766620 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:16.766629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.767780 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.767785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:16.768778 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:16.768791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.769583 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.769590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:16.769775 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:16.770401 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:16.770419 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:16.773770 (+ 3351us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
W20260929 02:56:16.774247 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:16.765240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:43467 (local address 127.15.18.126:45171)
0929 02:56:16.765393 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:16.765397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:16.765712 (+ 315us) server_negotiation.cc:407] Connection header received
0929 02:56:16.765865 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:16.765868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:16.765933 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:16.766024 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:16.766786 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.767613 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.768930 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.769411 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.769533 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:16.774007 (+ 4474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:43467: BlockingRecv error: recv got EOF from 127.15.18.65:43467 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 02:56:16 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:16 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:16.983911 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:16.974543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:34789)
0929 02:56:16.975120 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:16.975136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:16.975258 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:16.975623 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:16.975626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:16.975998 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:16.976178 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:16.976187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.977499 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.977504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:16.978692 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:16.978708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.979500 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.979527 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:16.979662 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:16.980272 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:16.980291 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:16.983658 (+ 3367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":315,"threads_started":1}
W20260929 02:56:16.984061 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:16.974649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:34789 (local address 127.15.18.126:45171)
0929 02:56:16.974805 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:16.974810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:16.975238 (+ 428us) server_negotiation.cc:407] Connection header received
0929 02:56:16.975359 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:16.975363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:16.975429 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:16.975545 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:16.976334 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.977342 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.978840 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:16.979308 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:16.979406 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:16.983867 (+ 4461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:34789: BlockingRecv error: recv got EOF from 127.15.18.67:34789 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 02:56:17 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:17 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:17.378782 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:17.365527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:52213)
0929 02:56:17.366176 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:17.366192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:17.366330 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:17.366647 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:17.366651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:17.367066 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:17.367216 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:17.367224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.368234 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.368240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:17.369180 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:17.369192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.369881 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.369887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:17.370058 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:17.375056 (+ 4998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:17.375076 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:17.378520 (+ 3444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":419,"threads_started":1}
W20260929 02:56:17.378950 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:17.365669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:52213 (local address 127.15.18.126:45171)
0929 02:56:17.365868 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:17.365875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:17.366292 (+ 417us) server_negotiation.cc:407] Connection header received
0929 02:56:17.366451 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:17.366455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:17.366516 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:17.366596 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:17.367357 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.368095 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.369325 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.369739 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.369846 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:17.378742 (+ 8896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:52213: BlockingRecv error: recv got EOF from 127.15.18.66:52213 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 02:56:17 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:17 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:17.786048 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:17.775137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:48963)
0929 02:56:17.775711 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:17.775733 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:17.775888 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:17.776467 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:17.776472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:17.777027 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:17.777254 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:17.777267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.778593 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.778668 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:17.780262 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:17.780281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.781080 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.781087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:17.781276 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:17.781923 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:17.781948 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:17.785793 (+ 3845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":376,"threads_started":1}
W20260929 02:56:17.786230 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:17.775431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:48963 (local address 127.15.18.126:45171)
0929 02:56:17.775620 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:17.775626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:17.775819 (+ 193us) server_negotiation.cc:407] Connection header received
0929 02:56:17.776066 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:17.776072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:17.776157 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:17.776233 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:17.777381 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.778304 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.780404 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.780893 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.781012 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:17.786009 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:48963: BlockingRecv error: recv got EOF from 127.15.18.65:48963 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 02:56:17 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:17 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:17.993700 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:17.984810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:50481)
0929 02:56:17.985289 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:17.985305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:17.985419 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:17.985790 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:17.985794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:17.986282 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:17.986468 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:17.986481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.987701 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.987707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:17.988699 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:17.988711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.989478 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.989482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:17.989618 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:17.990256 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:17.990275 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:17.993480 (+ 3205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
W20260929 02:56:17.993991 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:17.984912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:50481 (local address 127.15.18.126:45171)
0929 02:56:17.985186 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:17.985193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:17.985402 (+ 209us) server_negotiation.cc:407] Connection header received
0929 02:56:17.985531 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:17.985535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:17.985622 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:17.985729 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:17.986650 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.987569 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.988834 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:17.989338 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:17.989433 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:17.993717 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:50481: BlockingRecv error: recv got EOF from 127.15.18.67:50481 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 29 02:56:18 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:18 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:18.388338 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:18.379757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:54669)
0929 02:56:18.380287 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:18.380306 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:18.380443 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:18.380764 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:18.380767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:18.381164 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:18.381322 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:18.381334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.382420 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.382424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:18.383478 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:18.383490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.384164 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.384169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:18.384283 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:18.384821 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:18.384838 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:18.388061 (+ 3223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":311,"threads_started":1}
W20260929 02:56:18.388439 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:18.379798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:54669 (local address 127.15.18.126:45171)
0929 02:56:18.380001 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:18.380008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:18.380388 (+ 380us) server_negotiation.cc:407] Connection header received
0929 02:56:18.380559 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:18.380564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:18.380636 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:18.380727 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:18.381457 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.382289 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.383619 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.384041 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.384128 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:18.388267 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:54669: BlockingRecv error: recv got EOF from 127.15.18.66:54669 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 02:56:18 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:18 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:18.796134 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:18.786952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:48099)
0929 02:56:18.787539 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:18.787560 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:18.787713 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:18.788059 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:18.788062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:18.788530 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:18.788741 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:18.788751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.789840 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.789844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:18.790800 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:18.790813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.791531 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.791536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:18.791697 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:18.792270 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:18.792287 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:18.795899 (+ 3612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":286,"threads_started":1}
W20260929 02:56:18.796247 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:18.787206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:48099 (local address 127.15.18.126:45171)
0929 02:56:18.787430 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:18.787434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:18.787679 (+ 245us) server_negotiation.cc:407] Connection header received
0929 02:56:18.787861 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:18.787866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:18.787930 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:18.788014 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:18.788901 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.789711 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.790957 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.791389 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.791497 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:18.796041 (+ 4544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:48099: BlockingRecv error: recv got EOF from 127.15.18.65:48099 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 29 02:56:18 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:18 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:19.003647 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:18.994584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:35575)
0929 02:56:18.995081 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:18.995101 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:18.995226 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:18.995613 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:18.995619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:18.996001 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:18.996175 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:18.996185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.997222 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.997226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:18.998262 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:18.998276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.999049 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.999061 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:18.999192 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:18.999834 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:18.999854 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:19.003295 (+ 3441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":274,"threads_started":1}
W20260929 02:56:19.003799 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:18.994705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:35575 (local address 127.15.18.126:45171)
0929 02:56:18.994873 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:18.994877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:18.995211 (+ 334us) server_negotiation.cc:407] Connection header received
0929 02:56:18.995366 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:18.995370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:18.995434 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:18.995551 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:18.996325 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.997082 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.998429 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:18.998874 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:18.998982 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:19.003542 (+ 4560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:35575: BlockingRecv error: recv got EOF from 127.15.18.67:35575 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 02:56:19 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:19 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:19.398448 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:19.389278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:45925)
0929 02:56:19.389753 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:19.389768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:19.389884 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:19.390247 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:19.390251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:19.390735 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:19.390931 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:19.390941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:19.392033 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:19.392037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:19.392961 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:19.392972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:19.393686 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:19.393691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:19.393834 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:19.394419 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:19.394438 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:19.398130 (+ 3692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":290,"threads_started":1}
W20260929 02:56:19.398535 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:19.389373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:45925 (local address 127.15.18.126:45171)
0929 02:56:19.389594 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:19.389601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:19.389866 (+ 265us) server_negotiation.cc:407] Connection header received
0929 02:56:19.390001 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:19.390004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:19.390068 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:19.390156 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:19.391067 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:19.391891 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:19.393089 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:19.393525 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:19.393628 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:19.398309 (+ 4681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:45925: BlockingRecv error: recv got EOF from 127.15.18.66:45925 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 02:56:19 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:19 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:19.805853 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:19.797112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:40743)
0929 02:56:19.797632 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:19.797648 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:19.797765 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:19.798028 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:19.798031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:19.798456 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:19.798657 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:19.798666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:19.799785 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:19.799789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:19.800758 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:19.800771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:19.801448 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:19.801452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:19.801614 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:19.802219 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:19.802238 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:19.805558 (+ 3320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":280,"threads_started":1}
W20260929 02:56:19.806025 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:19.797226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:40743 (local address 127.15.18.126:45171)
0929 02:56:19.797459 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:19.797464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:19.797726 (+ 262us) server_negotiation.cc:407] Connection header received
0929 02:56:19.797858 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:19.797862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:19.797923 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:19.798002 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:19.798815 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:19.799630 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:19.800897 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:19.801316 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:19.801412 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:19.805779 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:40743: BlockingRecv error: recv got EOF from 127.15.18.65:40743 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 02:56:20 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:20 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:20.013826 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:20.004782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:47387)
0929 02:56:20.005250 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:20.005267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:20.005386 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:20.005667 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:20.005670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:20.006061 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:20.006250 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:20.006260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.007330 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.007334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:20.008314 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:20.008329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.009117 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.009122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:20.009280 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:20.009895 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:20.009913 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:20.013592 (+ 3679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":274,"threads_started":1}
W20260929 02:56:20.014139 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:20.004904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:47387 (local address 127.15.18.126:45171)
0929 02:56:20.005134 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:20.005140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:20.005411 (+ 271us) server_negotiation.cc:407] Connection header received
0929 02:56:20.005473 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:20.005476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:20.005556 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:20.005641 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:20.006409 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.007178 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.008472 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.008942 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.009049 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:20.013881 (+ 4832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:47387: BlockingRecv error: recv got EOF from 127.15.18.67:47387 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 02:56:20 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:20 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:20.408972 24696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:20.399599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:44271)
0929 02:56:20.400201 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:20.400218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:20.400354 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:20.400699 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:20.400704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:20.401128 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:20.401308 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:20.401318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.402564 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.402568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:20.403584 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:20.403597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.404343 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.404350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:20.404524 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:20.405161 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:20.405180 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:20.408717 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":339,"threads_started":1}
W20260929 02:56:20.409175 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:20.399634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:44271 (local address 127.15.18.126:45171)
0929 02:56:20.399884 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:20.399890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:20.400308 (+ 418us) server_negotiation.cc:407] Connection header received
0929 02:56:20.400467 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:20.400471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:20.400563 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:20.400644 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:20.401522 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.402393 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.403729 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.404155 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.404273 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:20.408957 (+ 4684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:44271: BlockingRecv error: recv got EOF from 127.15.18.66:44271 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 02:56:20 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:20 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:20.815635 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:20.806916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:34601)
0929 02:56:20.807482 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:20.807499 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:20.807654 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:20.807960 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:20.807965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:20.808369 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:20.808518 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:20.808526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.809517 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.809522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:20.810406 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:20.810418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.811121 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.811127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:20.811286 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:20.811847 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:20.811863 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:20.815229 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":335,"threads_started":1}
W20260929 02:56:20.815764 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:20.807068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:34601 (local address 127.15.18.126:45171)
0929 02:56:20.807272 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:20.807278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:20.807599 (+ 321us) server_negotiation.cc:407] Connection header received
0929 02:56:20.807768 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:20.807771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:20.807838 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:20.807918 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:20.808657 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.809378 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.810541 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:20.810976 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:20.811089 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:20.815469 (+ 4380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:34601: BlockingRecv error: recv got EOF from 127.15.18.65:34601 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 02:56:21 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:21 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:21.023353 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:21.014860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:37175)
0929 02:56:21.015412 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:21.015429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:21.015581 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:21.015876 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:21.015879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:21.016267 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:21.016433 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:21.016442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.017488 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.017492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:21.018416 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:21.018427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.019150 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.019156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:21.019284 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:21.019893 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:21.019910 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:21.023104 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":307,"threads_started":1}
W20260929 02:56:21.023500 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:21.014878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:37175 (local address 127.15.18.126:45171)
0929 02:56:21.015070 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:21.015076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:21.015578 (+ 502us) server_negotiation.cc:407] Connection header received
0929 02:56:21.015677 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:21.015681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:21.015744 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:21.015825 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:21.016602 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.017350 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.018581 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.018985 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.019084 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:21.023315 (+ 4231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:37175: BlockingRecv error: recv got EOF from 127.15.18.67:37175 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 02:56:21 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:21 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:21.418540 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:21.409944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:58709)
0929 02:56:21.410419 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:21.410435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:21.410550 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:21.410838 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:21.410841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:21.411221 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:21.411381 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:21.411389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.412469 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.412473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:21.413409 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:21.413422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.414190 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.414194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:21.414313 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:21.414861 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:21.414879 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:21.418305 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":264,"threads_started":1}
W20260929 02:56:21.418814 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:21.410124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:58709 (local address 127.15.18.126:45171)
0929 02:56:21.410289 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:21.410295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:21.410507 (+ 212us) server_negotiation.cc:407] Connection header received
0929 02:56:21.410654 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:21.410657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:21.410733 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:21.410808 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:21.411584 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.412318 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.413569 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.414053 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.414167 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:21.418523 (+ 4356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:58709: BlockingRecv error: recv got EOF from 127.15.18.66:58709 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 02:56:21 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:21 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:21.825519 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:21.816959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:56743)
0929 02:56:21.817520 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:21.817536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:21.817677 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:21.817987 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:21.817991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:21.818376 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:21.818559 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:21.818568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.819616 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.819622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:21.820532 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:21.820544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.821221 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.821226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:21.821393 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:21.822016 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:21.822055 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:21.825289 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":325,"threads_started":1}
W20260929 02:56:21.825645 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:21.817099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:56743 (local address 127.15.18.126:45171)
0929 02:56:21.817306 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:21.817312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:21.817625 (+ 313us) server_negotiation.cc:407] Connection header received
0929 02:56:21.817797 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:21.817801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:21.817860 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:21.817937 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:21.818740 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.819447 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.820681 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:21.821072 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:21.821184 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:21.825451 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:56743: BlockingRecv error: recv got EOF from 127.15.18.65:56743 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 02:56:22 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:22 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.67@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:22.033303 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:22.024424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.67:57777)
0929 02:56:22.024882 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:22.024898 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:22.025018 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:22.025299 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:22.025302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:22.025678 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:22.025830 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:22.025839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.026906 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.026911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:22.027949 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:22.027964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.028661 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.028666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:22.028786 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:22.029412 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:22.029431 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:22.033058 (+ 3627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":253,"threads_started":1}
W20260929 02:56:22.033506 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:22.024538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:57777 (local address 127.15.18.126:45171)
0929 02:56:22.024856 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:22.024862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:22.024969 (+ 107us) server_negotiation.cc:407] Connection header received
0929 02:56:22.025116 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:22.025120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:22.025186 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:22.025264 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:22.025986 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.026747 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.028090 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.028521 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.028642 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:22.033281 (+ 4639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.67:57777: BlockingRecv error: recv got EOF from 127.15.18.67:57777 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Sep 29 02:56:22 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:22 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.66@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:22.434620 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:22.419867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.66:59239)
0929 02:56:22.420370 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:22.420386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:22.420501 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:22.420930 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:22.420936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:22.421389 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:22.421645 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:22.421661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.422944 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.422948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:22.424039 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:22.424053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.424751 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.424756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:22.424896 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:22.430896 (+ 6000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:22.430916 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:22.434306 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
W20260929 02:56:22.434760 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:22.420035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:59239 (local address 127.15.18.126:45171)
0929 02:56:22.420222 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:22.420227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:22.420488 (+ 261us) server_negotiation.cc:407] Connection header received
0929 02:56:22.420630 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:22.420634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:22.420735 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:22.420826 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:22.421822 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.422803 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.424189 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.424619 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.424720 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:22.434508 (+ 9788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.66:59239: BlockingRecv error: recv got EOF from 127.15.18.66:59239 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 02:56:22 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:56:22 dist-test-slave-q5f0 krb5kdc[23656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.18.65@KRBTEST.COM for oryx/127.15.18.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:56:22.835868 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:22.826670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45171 (local address 127.15.18.65:40639)
0929 02:56:22.827233 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:22.827248 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:22.827379 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:22.827755 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:22.827760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:22.828165 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:22.828334 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:22.828343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.829360 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.829366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:22.830312 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:22.830324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.831035 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.831041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:22.831206 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:22.831912 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:22.831932 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:22.835498 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.18.126:45171: Server oryx/127.15.18.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":309,"threads_started":1}
W20260929 02:56:22.836079 24392 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:22.826772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:40639 (local address 127.15.18.126:45171)
0929 02:56:22.827008 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:22.827015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:22.827338 (+ 323us) server_negotiation.cc:407] Connection header received
0929 02:56:22.827535 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:22.827540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:22.827625 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:22.827705 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:22.828473 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.829213 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.830449 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.830889 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.830995 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:22.835805 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.18.65:40639: BlockingRecv error: recv got EOF from 127.15.18.65:40639 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:56:22.862944 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24243
I20260929 02:56:22.875566 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:22.864156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.15.18.65:34823)
0929 02:56:22.864755 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:22.864764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:22.864794 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:56:22.864858 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:22.864864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:22.865094 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:22.865256 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:22.866744 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.867917 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.869220 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.869873 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.869991 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:22.872453 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:22.872463 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:22.872477 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:22.872518 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:22.875013 (+ 2495us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.18.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:56:22.875296 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33200: Request ticket server kudu/127.15.18.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":247,"threads_started":1}
W20260929 02:56:22.875669 24705 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33200: Request ticket server kudu/127.15.18.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:56:22.875694 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24243: Not authorized: Client connection negotiation failed: client connection to 127.15.18.65:34823: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.18.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:56:22.875895 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 24243
I20260929 02:56:22.889820 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24394
I20260929 02:56:22.902647 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:22.891034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.15.18.66:40611)
0929 02:56:22.891720 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:22.891729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:22.891766 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:56:22.891845 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:22.891853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:22.892102 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:22.892276 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:22.893709 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.894810 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.896246 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.896925 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.897098 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:22.899559 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:22.899572 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:22.899592 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:22.899650 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:22.902128 (+ 2478us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.18.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:56:22.902404 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55770: Request ticket server kudu/127.15.18.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":336,"threads_started":1}
W20260929 02:56:22.902740 24706 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55770: Request ticket server kudu/127.15.18.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:56:22.902822 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24394: Not authorized: Client connection negotiation failed: client connection to 127.15.18.66:40611: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.18.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:56:22.902961 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 24394
I20260929 02:56:22.915864 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24527
I20260929 02:56:22.928133 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:22.916996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.15.18.67:39271)
0929 02:56:22.917575 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:22.917582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:22.917607 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:56:22.917670 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:22.917676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:22.917941 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:22.918104 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:22.919899 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.920958 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.922097 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:22.922634 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:22.922764 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:22.925170 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:22.925178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:22.925192 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:22.925234 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:22.927606 (+ 2372us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.18.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:56:22.927895 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57310: Request ticket server kudu/127.15.18.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":292,"threads_started":1}
W20260929 02:56:22.928187 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24527: Not authorized: Client connection negotiation failed: client connection to 127.15.18.67:39271: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.18.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:56:22.928227 24707 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57310: Request ticket server kudu/127.15.18.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:56:22.928331 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 24527
I20260929 02:56:22.941105 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24174
I20260929 02:56:22.942334 24236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:23.055732 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 24174
2026-09-29T02:56:23Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21137 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T02:56:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:56:23Z Disabled control of system clock
I20260929 02:56:23.125476 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:40795
--webserver_interface=127.15.18.126
--webserver_port=0
--builtin_ntp_servers=127.15.18.84:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:40795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 02:56:23.451974 24712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:23.452277 24712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:23.452375 24712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:23.459717 24712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:56:23.459836 24712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:23.459892 24712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:23.459971 24712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:56:23.460024 24712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:56:23.469228 24712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:40795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:40795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:23.470542 24712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:23.472292 24712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:23.484107 24717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:23.484126 24712 server_base.cc:1034] running on GCE node
W20260929 02:56:23.484320 24718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:23.484089 24720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:23.485246 24712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:23.486671 24712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:23.487879 24712 hybrid_clock.cc:648] HybridClock initialized: now 1790650583487831 us; error 36 us; skew 500 ppm
I20260929 02:56:23.488210 24712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:56:23.490455 24712 webserver.cc:460] Webserver started at http://127.15.18.126:37901/ using document root <none> and password file <none>
I20260929 02:56:23.491129 24712 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:23.491225 24712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:23.491564 24712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:23.494388 24712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "ae511908d8bf49eaad30775821eff68a"
format_stamp: "Formatted at 2026-09-29 02:56:23 on dist-test-slave-q5f0"
server_key: "2674b20a1ea8808379c0118c52a48249"
server_key_iv: "cdeb3913e0ebdc518b34f95b2d58c679"
server_key_version: "encryptionkey@0"
I20260929 02:56:23.495157 24712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "ae511908d8bf49eaad30775821eff68a"
format_stamp: "Formatted at 2026-09-29 02:56:23 on dist-test-slave-q5f0"
server_key: "2674b20a1ea8808379c0118c52a48249"
server_key_iv: "cdeb3913e0ebdc518b34f95b2d58c679"
server_key_version: "encryptionkey@0"
I20260929 02:56:23.500272 24712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 02:56:23.503720 24726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:23.505074 24712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:56:23.505234 24712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "ae511908d8bf49eaad30775821eff68a"
format_stamp: "Formatted at 2026-09-29 02:56:23 on dist-test-slave-q5f0"
server_key: "2674b20a1ea8808379c0118c52a48249"
server_key_iv: "cdeb3913e0ebdc518b34f95b2d58c679"
server_key_version: "encryptionkey@0"
I20260929 02:56:23.505427 24712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:23.517814 24712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:23.518646 24712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:23.518923 24712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:23.536303 24712 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:40795
I20260929 02:56:23.536326 24777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:40795 every 8 connection(s)
I20260929 02:56:23.537637 24712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:56:23.541662 24778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:56:23.546617 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 24712
I20260929 02:56:23.546803 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:56:23.547575 15433 external_mini_cluster.cc:1428] Setting key 0c5e98203482aaa953ea3ba6788ea863
I20260929 02:56:23.553162 24778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a: Bootstrap starting.
I20260929 02:56:23.556303 24778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a: Neither blocks nor log segments found. Creating new log.
I20260929 02:56:23.557336 24778 log.cc:824] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:56:23.560346 24778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a: No bootstrap required, opened a new log
I20260929 02:56:23.561337 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:23.549880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.15.18.126:40795)
0929 02:56:23.550919 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:23.550938 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:23.550979 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:56:23.552626 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:23.552650 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:23.553134 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:23.553530 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:23.554163 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:23.555480 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:23.556892 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:23.557609 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:23.557791 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:23.558986 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:23.558993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:23.559009 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:23.559606 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:23.559692 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:23.559714 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:23.559827 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:23.560033 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 02:56:23.560359 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":319,"threads_started":1}
I20260929 02:56:23.568221 24778 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae511908d8bf49eaad30775821eff68a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40795 } }
I20260929 02:56:23.568596 24778 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:56:23.568688 24778 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae511908d8bf49eaad30775821eff68a, State: Initialized, Role: FOLLOWER
I20260929 02:56:23.569288 24778 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae511908d8bf49eaad30775821eff68a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40795 } }
I20260929 02:56:23.569504 24778 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:56:23.569633 24778 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:56:23.569792 24778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:56:23.571991 24778 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae511908d8bf49eaad30775821eff68a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40795 } }
I20260929 02:56:23.572559 24778 leader_election.cc:302] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae511908d8bf49eaad30775821eff68a; no voters:
I20260929 02:56:23.573155 24778 leader_election.cc:288] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:56:23.573283 24783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:56:23.573915 24783 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [term 1 LEADER]: Becoming Leader. State: Replica: ae511908d8bf49eaad30775821eff68a, State: Running, Role: LEADER
I20260929 02:56:23.574536 24783 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae511908d8bf49eaad30775821eff68a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40795 } }
I20260929 02:56:23.575369 24778 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:56:23.577476 24785 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae511908d8bf49eaad30775821eff68a. Latest consensus state: current_term: 1 leader_uuid: "ae511908d8bf49eaad30775821eff68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae511908d8bf49eaad30775821eff68a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40795 } } }
I20260929 02:56:23.577471 24784 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae511908d8bf49eaad30775821eff68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae511908d8bf49eaad30775821eff68a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40795 } } }
I20260929 02:56:23.577888 24785 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [sys.catalog]: This master's current role is: LEADER
I20260929 02:56:23.577893 24784 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a [sys.catalog]: This master's current role is: LEADER
I20260929 02:56:23.580514 24790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:56:23.585701 24790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:56:23.595489 24790 catalog_manager.cc:1347] Generated new cluster ID: d99a8a38d14549b09c6a441318207bfd
I20260929 02:56:23.595645 24790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:56:23.613529 24790 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:56:23.614816 24790 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:56:23.635815 24790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae511908d8bf49eaad30775821eff68a: Generated new TSK 0
I20260929 02:56:23.636751 24790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:56:23.649046 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:40795
--builtin_ntp_servers=127.15.18.84:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 02:56:23.985011 24802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:23.985301 24802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:23.985410 24802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:23.992878 24802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:23.993000 24802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:23.993131 24802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:56:24.002491 24802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:40795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:24.003822 24802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:24.005631 24802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:24.018640 24807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:24.018689 24810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:24.018703 24808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:24.019405 24802 server_base.cc:1034] running on GCE node
I20260929 02:56:24.020098 24802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:24.021175 24802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:24.022410 24802 hybrid_clock.cc:648] HybridClock initialized: now 1790650584022348 us; error 50 us; skew 500 ppm
I20260929 02:56:24.022738 24802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:56:24.025133 24802 webserver.cc:460] Webserver started at http://127.15.18.65:35951/ using document root <none> and password file <none>
I20260929 02:56:24.025846 24802 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:24.025935 24802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:24.026213 24802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:24.029143 24802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "161d5cb6e78e47f797b6da2fbd6c3c28"
format_stamp: "Formatted at 2026-09-29 02:56:24 on dist-test-slave-q5f0"
server_key: "8a746dc4416bc26e6ab03fc66b9133e2"
server_key_iv: "fc7da7be7beb8565a2a8a1f96ab9699b"
server_key_version: "encryptionkey@0"
I20260929 02:56:24.029910 24802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "161d5cb6e78e47f797b6da2fbd6c3c28"
format_stamp: "Formatted at 2026-09-29 02:56:24 on dist-test-slave-q5f0"
server_key: "8a746dc4416bc26e6ab03fc66b9133e2"
server_key_iv: "fc7da7be7beb8565a2a8a1f96ab9699b"
server_key_version: "encryptionkey@0"
I20260929 02:56:24.035235 24802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:56:24.038982 24816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:24.040632 24802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:56:24.040794 24802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "161d5cb6e78e47f797b6da2fbd6c3c28"
format_stamp: "Formatted at 2026-09-29 02:56:24 on dist-test-slave-q5f0"
server_key: "8a746dc4416bc26e6ab03fc66b9133e2"
server_key_iv: "fc7da7be7beb8565a2a8a1f96ab9699b"
server_key_version: "encryptionkey@0"
I20260929 02:56:24.040998 24802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:24.070719 24802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:24.071630 24802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:24.071894 24802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:24.073029 24802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:24.074699 24802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:24.074786 24802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:24.074872 24802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:24.074923 24802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:24.105314 24802 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:43159
I20260929 02:56:24.105370 24928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:43159 every 8 connection(s)
I20260929 02:56:24.106626 24802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:56:24.114013 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 24802
I20260929 02:56:24.114207 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:56:24.114965 15433 external_mini_cluster.cc:1428] Setting key a05e47ee6b41e844409a15ec41bb19c8
I20260929 02:56:24.117803 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:40795
--builtin_ntp_servers=127.15.18.84:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 02:56:24.119567 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:24.109333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:45065 (local address 127.15.18.126:40795)
0929 02:56:24.109782 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:24.109790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:24.110947 (+ 1157us) server_negotiation.cc:407] Connection header received
0929 02:56:24.112629 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:24.112633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:24.112699 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:24.112796 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:24.113794 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:24.114866 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:24.116755 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:24.117189 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:24.117295 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:24.118853 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:24.118858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:24.118862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:24.118878 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:24.118922 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:24.118935 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:24.119013 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:24.119222 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 02:56:24.119296 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":219,"threads_started":1}
I20260929 02:56:24.120918 24932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:24.109932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:40795 (local address 127.15.18.65:45065)
0929 02:56:24.110797 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:24.110838 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:24.112401 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:24.112974 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:24.112982 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:24.113085 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:24.113638 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:24.113654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:24.115030 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:24.115036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:24.116590 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:24.116606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:24.117431 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:24.117440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:24.117773 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:24.118482 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:56:24.118486 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:24.118540 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:56:24.118653 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:24.119027 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:24.119037 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:24.119045 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:56:24.119221 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 02:56:24.119643 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":248,"threads_started":1}
I20260929 02:56:24.122568 24929 heartbeater.cc:347] Connected to a master server at 127.15.18.126:40795
I20260929 02:56:24.122843 24929 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:24.123633 24929 heartbeater.cc:511] Master 127.15.18.126:40795 requested a full tablet report, sending...
I20260929 02:56:24.125639 24743 ts_manager.cc:194] Registered new tserver with Master: 161d5cb6e78e47f797b6da2fbd6c3c28 (127.15.18.65:43159)
I20260929 02:56:24.127105 24743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.65:45065
W20260929 02:56:24.442009 24933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:24.442309 24933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:24.442421 24933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:24.449870 24933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:24.449991 24933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:24.450139 24933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:56:24.459419 24933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:40795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:24.460731 24933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:24.462477 24933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:24.475191 24938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:24.475205 24941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:24.475238 24939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:24.475986 24933 server_base.cc:1034] running on GCE node
I20260929 02:56:24.476611 24933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:24.477541 24933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:24.478775 24933 hybrid_clock.cc:648] HybridClock initialized: now 1790650584478710 us; error 46 us; skew 500 ppm
I20260929 02:56:24.479094 24933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:56:24.481439 24933 webserver.cc:460] Webserver started at http://127.15.18.66:46693/ using document root <none> and password file <none>
I20260929 02:56:24.482108 24933 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:24.482197 24933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:24.482473 24933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:24.485303 24933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "3f15fa015607451b82bd61790c1d7a8c"
format_stamp: "Formatted at 2026-09-29 02:56:24 on dist-test-slave-q5f0"
server_key: "21cebe194fdaa12693ced56dd91e4013"
server_key_iv: "94e9cc38ecb96baac18202d9c37c722b"
server_key_version: "encryptionkey@0"
I20260929 02:56:24.486047 24933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "3f15fa015607451b82bd61790c1d7a8c"
format_stamp: "Formatted at 2026-09-29 02:56:24 on dist-test-slave-q5f0"
server_key: "21cebe194fdaa12693ced56dd91e4013"
server_key_iv: "94e9cc38ecb96baac18202d9c37c722b"
server_key_version: "encryptionkey@0"
I20260929 02:56:24.491233 24933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:56:24.495137 24947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:24.496730 24933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:56:24.496896 24933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "3f15fa015607451b82bd61790c1d7a8c"
format_stamp: "Formatted at 2026-09-29 02:56:24 on dist-test-slave-q5f0"
server_key: "21cebe194fdaa12693ced56dd91e4013"
server_key_iv: "94e9cc38ecb96baac18202d9c37c722b"
server_key_version: "encryptionkey@0"
I20260929 02:56:24.497116 24933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:24.531771 24933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:24.532678 24933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:24.532959 24933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:24.533907 24933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:24.535375 24933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:24.535468 24933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:24.535583 24933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:24.535650 24933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:24.566437 24933 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:45109
I20260929 02:56:24.566491 25059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:45109 every 8 connection(s)
I20260929 02:56:24.567783 24933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:56:24.570513 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 24933
I20260929 02:56:24.570647 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:56:24.571074 15433 external_mini_cluster.cc:1428] Setting key 0be4943365f08b0cb9e4ff47f3346a39
I20260929 02:56:24.573714 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:40795
--builtin_ntp_servers=127.15.18.84:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 02:56:24.580392 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:24.570205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:49521 (local address 127.15.18.126:40795)
0929 02:56:24.570415 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:24.570421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:24.571964 (+ 1543us) server_negotiation.cc:407] Connection header received
0929 02:56:24.573812 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:24.573818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:24.573899 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:24.573995 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:24.575007 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:24.576047 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:24.577645 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:24.578157 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:24.578274 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:24.579612 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:24.579617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:24.579622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:24.579653 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:24.579689 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:24.579702 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:24.579793 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:24.580045 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 02:56:24.580124 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:56:24.581794 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:24.570793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:40795 (local address 127.15.18.66:49521)
0929 02:56:24.571816 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:24.571859 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:24.573560 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:24.574184 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:24.574193 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:24.574297 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:24.574857 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:24.574874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:24.576199 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:24.576204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:24.577508 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:24.577520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:24.578383 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:24.578389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:24.578621 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:24.579293 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:56:24.579295 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:24.579339 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:56:24.579423 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:24.579801 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:24.579812 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:24.579821 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:56:24.580024 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 02:56:24.580464 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":283,"threads_started":1}
I20260929 02:56:24.583227 25060 heartbeater.cc:347] Connected to a master server at 127.15.18.126:40795
I20260929 02:56:24.583549 25060 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:24.584362 25060 heartbeater.cc:511] Master 127.15.18.126:40795 requested a full tablet report, sending...
I20260929 02:56:24.585743 24743 ts_manager.cc:194] Registered new tserver with Master: 3f15fa015607451b82bd61790c1d7a8c (127.15.18.66:45109)
I20260929 02:56:24.586500 24743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.66:49521
W20260929 02:56:24.896939 25063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:24.897212 25063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:24.897320 25063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:24.904697 25063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:24.904829 25063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:24.904961 25063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:56:24.914213 25063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:40795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:24.915531 25063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:24.917245 25063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:24.930011 25068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:24.929877 25069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:24.929970 25071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:24.930536 25063 server_base.cc:1034] running on GCE node
I20260929 02:56:24.931167 25063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:24.932178 25063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:24.933386 25063 hybrid_clock.cc:648] HybridClock initialized: now 1790650584933336 us; error 38 us; skew 500 ppm
I20260929 02:56:24.933723 25063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:56:24.936210 25063 webserver.cc:460] Webserver started at http://127.15.18.67:45985/ using document root <none> and password file <none>
I20260929 02:56:24.936899 25063 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:24.936992 25063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:24.937299 25063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:24.940079 25063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "7d798f478a2a47df9fbbd2617ea0a8e5"
format_stamp: "Formatted at 2026-09-29 02:56:24 on dist-test-slave-q5f0"
server_key: "21aaea36463861492853fb8a80bbce3e"
server_key_iv: "2cb26425f0708d635daeb2ceeb67d233"
server_key_version: "encryptionkey@0"
I20260929 02:56:24.940871 25063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "7d798f478a2a47df9fbbd2617ea0a8e5"
format_stamp: "Formatted at 2026-09-29 02:56:24 on dist-test-slave-q5f0"
server_key: "21aaea36463861492853fb8a80bbce3e"
server_key_iv: "2cb26425f0708d635daeb2ceeb67d233"
server_key_version: "encryptionkey@0"
I20260929 02:56:24.945839 25063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:56:24.949175 25077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:24.950610 25063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:56:24.950801 25063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "7d798f478a2a47df9fbbd2617ea0a8e5"
format_stamp: "Formatted at 2026-09-29 02:56:24 on dist-test-slave-q5f0"
server_key: "21aaea36463861492853fb8a80bbce3e"
server_key_iv: "2cb26425f0708d635daeb2ceeb67d233"
server_key_version: "encryptionkey@0"
I20260929 02:56:24.951004 25063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:24.977360 25063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:24.978251 25063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:24.978542 25063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:24.979537 25063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:24.981144 25063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:24.981246 25063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:24.981356 25063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:24.981422 25063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:25.011291 25063 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:34447
I20260929 02:56:25.011358 25189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:34447 every 8 connection(s)
I20260929 02:56:25.012596 25063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:56:25.017021 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 25063
I20260929 02:56:25.017189 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:56:25.017771 15433 external_mini_cluster.cc:1428] Setting key 0b80c01c6c124b630279d1a0aa91e414
I20260929 02:56:25.025696 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:25.015147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:43697 (local address 127.15.18.126:40795)
0929 02:56:25.015320 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:25.015324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:25.016946 (+ 1622us) server_negotiation.cc:407] Connection header received
0929 02:56:25.019224 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:25.019230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:25.019308 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:25.019397 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:25.020677 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:25.021413 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:25.022993 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:25.023442 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:25.023596 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:25.024925 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:25.024930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:25.024934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:25.024955 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:25.024991 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:25.025003 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:25.025084 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:25.025316 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 02:56:25.025404 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:56:25.026746 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:25.015742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:40795 (local address 127.15.18.67:43697)
0929 02:56:25.016726 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:25.016788 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:25.018909 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:25.019707 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:25.019716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:25.019832 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:25.020507 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:25.020525 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:25.021556 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:25.021560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:25.022838 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:25.022851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:25.023682 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:25.023687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:25.023901 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:25.024579 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:56:25.024581 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:25.024624 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:56:25.024718 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:25.025101 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:25.025110 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:25.025116 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:56:25.025295 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 02:56:25.025623 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":252,"threads_started":1}
I20260929 02:56:25.028178 25190 heartbeater.cc:347] Connected to a master server at 127.15.18.126:40795
I20260929 02:56:25.028501 25190 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:25.029372 25190 heartbeater.cc:511] Master 127.15.18.126:40795 requested a full tablet report, sending...
I20260929 02:56:25.030959 24743 ts_manager.cc:194] Registered new tserver with Master: 7d798f478a2a47df9fbbd2617ea0a8e5 (127.15.18.67:34447)
I20260929 02:56:25.031870 24743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.67:43697
I20260929 02:56:25.033622 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:56:25.039975 24931 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:25.038465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.15.18.126:40795)
0929 02:56:25.038671 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:25.038677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:25.038959 (+ 282us) server_negotiation.cc:407] Connection header received
0929 02:56:25.039119 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:25.039122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:25.039222 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:25.039300 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:25.039734 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48858: BlockingRecv error: recv got EOF from 127.0.0.1:48858 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:56:25.040788 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24802
I20260929 02:56:25.048339 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:25.041648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.15.18.65:43159)
0929 02:56:25.042152 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:25.042167 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:25.042193 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:56:25.042254 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:25.042260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:25.042464 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:25.042587 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:25.042955 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:25.043993 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:25.045263 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:25.045850 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:25.045965 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:25.046903 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:25.046911 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:25.046924 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:25.047548 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:25.047601 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:25.047633 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:25.047717 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:25.047877 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 02:56:25.047984 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":235,"threads_started":1}
I20260929 02:56:25.049511 24924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:25.130091 24929 heartbeater.cc:503] Master 127.15.18.126:40795 was elected leader, sending a full tablet report...
I20260929 02:56:25.132109 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 24802
I20260929 02:56:25.145238 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24933
I20260929 02:56:25.153101 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:25.146204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.15.18.66:45109)
0929 02:56:25.146699 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:25.146706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:25.146730 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:56:25.146788 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:25.146795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:25.147007 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:25.147161 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:25.147604 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:25.148595 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:25.149826 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:25.150384 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:25.150500 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:25.151639 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:25.151650 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:25.151668 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:25.152300 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:25.152348 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:25.152368 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:25.152470 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:25.152626 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 02:56:25.152761 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":236,"threads_started":1}
I20260929 02:56:25.154438 25055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:25.235349 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 24933
I20260929 02:56:25.248147 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 25063
I20260929 02:56:25.257061 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:25.249272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.15.18.67:34447)
0929 02:56:25.249960 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:25.249968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:25.250003 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:56:25.250077 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:25.250087 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:25.250352 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:25.250538 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:25.251084 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:25.252210 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:25.253530 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:25.254143 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:25.254263 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:25.255437 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:25.255448 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:25.255462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:25.256189 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:25.256246 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:25.256269 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:25.256389 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:25.256612 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 02:56:25.256736 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":378,"threads_started":1}
I20260929 02:56:25.258414 25185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:25.339496 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 25063
I20260929 02:56:25.353703 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 24712
I20260929 02:56:25.354982 24773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:25.455987 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 24712
2026-09-29T02:56:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2396 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T02:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:56:25Z Disabled control of system clock
I20260929 02:56:25.520403 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:36441
--webserver_interface=127.15.18.126
--webserver_port=0
--builtin_ntp_servers=127.15.18.84:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:36441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:56:25.836617 25209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:25.836905 25209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:25.837007 25209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:25.844244 25209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:56:25.844353 25209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:25.844395 25209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:25.844475 25209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:56:25.844533 25209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:56:25.853690 25209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:36441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:36441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:25.855013 25209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:25.856827 25209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:25.868315 25214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:25.868343 25217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:25.868404 25215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:25.868957 25209 server_base.cc:1034] running on GCE node
I20260929 02:56:25.869735 25209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:25.871101 25209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:25.872305 25209 hybrid_clock.cc:648] HybridClock initialized: now 1790650585872254 us; error 48 us; skew 500 ppm
I20260929 02:56:25.872676 25209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:56:25.874861 25209 webserver.cc:460] Webserver started at http://127.15.18.126:43415/ using document root <none> and password file <none>
I20260929 02:56:25.875569 25209 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:25.875679 25209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:25.875980 25209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:25.878800 25209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "1a3f9fda0aaf41de99ab23e836985fc3"
format_stamp: "Formatted at 2026-09-29 02:56:25 on dist-test-slave-q5f0"
server_key: "e2a4478f6dee6867e8d3a6cdfb38e040"
server_key_iv: "3e0d868b66f7880d151a7a12d5e605d7"
server_key_version: "encryptionkey@0"
I20260929 02:56:25.879649 25209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "1a3f9fda0aaf41de99ab23e836985fc3"
format_stamp: "Formatted at 2026-09-29 02:56:25 on dist-test-slave-q5f0"
server_key: "e2a4478f6dee6867e8d3a6cdfb38e040"
server_key_iv: "3e0d868b66f7880d151a7a12d5e605d7"
server_key_version: "encryptionkey@0"
I20260929 02:56:25.884694 25209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:56:25.888123 25223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:25.889672 25209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:56:25.889844 25209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "1a3f9fda0aaf41de99ab23e836985fc3"
format_stamp: "Formatted at 2026-09-29 02:56:25 on dist-test-slave-q5f0"
server_key: "e2a4478f6dee6867e8d3a6cdfb38e040"
server_key_iv: "3e0d868b66f7880d151a7a12d5e605d7"
server_key_version: "encryptionkey@0"
I20260929 02:56:25.890053 25209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:25.896090 25209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:25.896951 25209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:25.897246 25209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:25.914523 25209 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:36441
I20260929 02:56:25.914534 25274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:36441 every 8 connection(s)
I20260929 02:56:25.915962 25209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:56:25.919775 25275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:56:25.920457 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 25209
I20260929 02:56:25.920641 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:56:25.921221 15433 external_mini_cluster.cc:1428] Setting key c88e6da547c4424dc2f98ce7d112ca6a
I20260929 02:56:25.929306 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:25.923143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.15.18.126:36441)
0929 02:56:25.924179 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:25.924190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:25.924234 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:56:25.925733 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:25.925755 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:25.926215 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:25.926566 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:25.927700 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:25.927707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:25.927721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:25.927735 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:25.927798 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:25.927814 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:25.927984 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:25.928089 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 02:56:25.928356 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":402,"threads_started":1}
I20260929 02:56:25.932998 25275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3: Bootstrap starting.
I20260929 02:56:25.935885 25275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3: Neither blocks nor log segments found. Creating new log.
I20260929 02:56:25.936816 25275 log.cc:824] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:56:25.939677 25275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3: No bootstrap required, opened a new log
I20260929 02:56:25.946869 25275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3f9fda0aaf41de99ab23e836985fc3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 36441 } }
I20260929 02:56:25.947278 25275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:56:25.947372 25275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a3f9fda0aaf41de99ab23e836985fc3, State: Initialized, Role: FOLLOWER
I20260929 02:56:25.947978 25275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3f9fda0aaf41de99ab23e836985fc3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 36441 } }
I20260929 02:56:25.948202 25275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:56:25.948333 25275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:56:25.948482 25275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:56:25.950665 25275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3f9fda0aaf41de99ab23e836985fc3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 36441 } }
I20260929 02:56:25.951265 25275 leader_election.cc:302] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a3f9fda0aaf41de99ab23e836985fc3; no voters:
I20260929 02:56:25.951943 25275 leader_election.cc:288] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:56:25.952095 25280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:56:25.952838 25280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 1a3f9fda0aaf41de99ab23e836985fc3, State: Running, Role: LEADER
I20260929 02:56:25.953461 25280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3f9fda0aaf41de99ab23e836985fc3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 36441 } }
I20260929 02:56:25.954174 25275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:56:25.956017 25282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a3f9fda0aaf41de99ab23e836985fc3. Latest consensus state: current_term: 1 leader_uuid: "1a3f9fda0aaf41de99ab23e836985fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3f9fda0aaf41de99ab23e836985fc3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 36441 } } }
I20260929 02:56:25.956420 25282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:56:25.960436 25288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:56:25.959750 25281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a3f9fda0aaf41de99ab23e836985fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3f9fda0aaf41de99ab23e836985fc3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 36441 } } }
I20260929 02:56:25.960729 25281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:56:25.966190 25288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:56:25.974614 25288 catalog_manager.cc:1347] Generated new cluster ID: b17b96cd88954715b8678b5150331fa6
I20260929 02:56:25.974740 25288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:56:26.006491 25288 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:56:26.006966 25288 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:56:26.013003 25288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a3f9fda0aaf41de99ab23e836985fc3: Generated new TSK 0
I20260929 02:56:26.013937 25288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:56:26.018841 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:36441
--builtin_ntp_servers=127.15.18.84:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:56:26.344758 25299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:26.345046 25299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:26.345163 25299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:26.352344 25299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:26.352442 25299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:26.352589 25299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:56:26.362099 25299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:36441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:26.363387 25299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:26.365199 25299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:26.379079 25304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:26.379103 25305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:26.379036 25307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:26.380017 25299 server_base.cc:1034] running on GCE node
I20260929 02:56:26.380641 25299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:26.381606 25299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:26.382812 25299 hybrid_clock.cc:648] HybridClock initialized: now 1790650586382760 us; error 42 us; skew 500 ppm
I20260929 02:56:26.383121 25299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:56:26.385424 25299 webserver.cc:460] Webserver started at http://127.15.18.65:45351/ using document root <none> and password file <none>
I20260929 02:56:26.386116 25299 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:26.386209 25299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:26.386515 25299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:26.391153 25299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "6d61223fbcd04f5dbaf589eef29d3b8d"
format_stamp: "Formatted at 2026-09-29 02:56:26 on dist-test-slave-q5f0"
server_key: "be1ac8a398b92fcfe20b100bcf3ebc6d"
server_key_iv: "7154b612dc7189a44fb2bc69f4ce61b4"
server_key_version: "encryptionkey@0"
I20260929 02:56:26.392000 25299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "6d61223fbcd04f5dbaf589eef29d3b8d"
format_stamp: "Formatted at 2026-09-29 02:56:26 on dist-test-slave-q5f0"
server_key: "be1ac8a398b92fcfe20b100bcf3ebc6d"
server_key_iv: "7154b612dc7189a44fb2bc69f4ce61b4"
server_key_version: "encryptionkey@0"
I20260929 02:56:26.397334 25299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:56:26.400803 25313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:26.402199 25299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:56:26.402328 25299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "6d61223fbcd04f5dbaf589eef29d3b8d"
format_stamp: "Formatted at 2026-09-29 02:56:26 on dist-test-slave-q5f0"
server_key: "be1ac8a398b92fcfe20b100bcf3ebc6d"
server_key_iv: "7154b612dc7189a44fb2bc69f4ce61b4"
server_key_version: "encryptionkey@0"
I20260929 02:56:26.402503 25299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:26.408015 25299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:26.408748 25299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:26.408963 25299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:26.409819 25299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:26.411144 25299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:26.411223 25299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:26.411302 25299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:26.411355 25299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:26.441126 25299 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:45705
I20260929 02:56:26.441197 25425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:45705 every 8 connection(s)
I20260929 02:56:26.442394 25299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:56:26.442845 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 25299
I20260929 02:56:26.443032 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:56:26.443848 15433 external_mini_cluster.cc:1428] Setting key 9430e289b29305e5c8213a21e5149647
I20260929 02:56:26.446899 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:36441
--builtin_ntp_servers=127.15.18.84:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 02:56:26.450583 25428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:26.445018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:35491 (local address 127.15.18.126:36441)
0929 02:56:26.445454 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:26.445459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:26.446772 (+ 1313us) server_negotiation.cc:407] Connection header received
0929 02:56:26.448459 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:26.448464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:26.448539 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:26.448628 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:26.449866 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:26.449872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:26.449883 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:26.449900 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:26.449938 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:26.449950 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:26.450046 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:26.450225 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 02:56:26.450289 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260929 02:56:26.451493 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:26.445728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:36441 (local address 127.15.18.65:35491)
0929 02:56:26.446637 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:26.446674 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:26.448220 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:26.448863 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:26.448871 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:26.448964 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:26.449590 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:56:26.449592 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:26.449629 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:56:26.449713 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:26.450038 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:26.450048 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:26.450055 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:56:26.450295 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 02:56:26.450558 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":289,"threads_started":1}
I20260929 02:56:26.453061 25426 heartbeater.cc:347] Connected to a master server at 127.15.18.126:36441
I20260929 02:56:26.453336 25426 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:26.453933 25426 heartbeater.cc:511] Master 127.15.18.126:36441 requested a full tablet report, sending...
I20260929 02:56:26.456058 25240 ts_manager.cc:194] Registered new tserver with Master: 6d61223fbcd04f5dbaf589eef29d3b8d (127.15.18.65:45705)
W20260929 02:56:26.763664 25430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:26.763944 25430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:26.764060 25430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:26.771318 25430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:26.771431 25430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:26.771601 25430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:56:26.780782 25430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:36441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:26.782071 25430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:26.783857 25430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:26.796849 25435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:26.796883 25438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:26.796919 25436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:26.798207 25430 server_base.cc:1034] running on GCE node
I20260929 02:56:26.798851 25430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:26.799824 25430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:26.801064 25430 hybrid_clock.cc:648] HybridClock initialized: now 1790650586801005 us; error 48 us; skew 500 ppm
I20260929 02:56:26.801379 25430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:56:26.803769 25430 webserver.cc:460] Webserver started at http://127.15.18.66:35187/ using document root <none> and password file <none>
I20260929 02:56:26.804453 25430 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:26.804561 25430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:26.804857 25430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:26.807816 25430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "5398afd204e648e9b7b992a88818f6e6"
format_stamp: "Formatted at 2026-09-29 02:56:26 on dist-test-slave-q5f0"
server_key: "73264b0c34c46c895e0fee5d324d1d7f"
server_key_iv: "19a8e2640203b94bbbf0e35f2adf3aa2"
server_key_version: "encryptionkey@0"
I20260929 02:56:26.808579 25430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "5398afd204e648e9b7b992a88818f6e6"
format_stamp: "Formatted at 2026-09-29 02:56:26 on dist-test-slave-q5f0"
server_key: "73264b0c34c46c895e0fee5d324d1d7f"
server_key_iv: "19a8e2640203b94bbbf0e35f2adf3aa2"
server_key_version: "encryptionkey@0"
I20260929 02:56:26.813696 25430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:56:26.817314 25444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:26.818764 25430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:56:26.818921 25430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "5398afd204e648e9b7b992a88818f6e6"
format_stamp: "Formatted at 2026-09-29 02:56:26 on dist-test-slave-q5f0"
server_key: "73264b0c34c46c895e0fee5d324d1d7f"
server_key_iv: "19a8e2640203b94bbbf0e35f2adf3aa2"
server_key_version: "encryptionkey@0"
I20260929 02:56:26.819128 25430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:26.824702 25430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:26.825469 25430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:26.825712 25430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:26.826620 25430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:26.828063 25430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:26.828161 25430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:26.828264 25430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:26.828333 25430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:26.858268 25430 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:45981
I20260929 02:56:26.858314 25556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:45981 every 8 connection(s)
I20260929 02:56:26.859617 25430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:56:26.867398 25428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:26.861995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:56979 (local address 127.15.18.126:36441)
0929 02:56:26.862172 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:26.862177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:26.863498 (+ 1321us) server_negotiation.cc:407] Connection header received
0929 02:56:26.865215 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:26.865220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:26.865292 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:26.865377 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:26.866683 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:26.866689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:26.866692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:26.866710 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:26.866740 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:26.866752 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:26.866853 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:26.867030 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 02:56:26.867110 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:56:26.868376 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:26.862512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:36441 (local address 127.15.18.66:56979)
0929 02:56:26.863353 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:26.863387 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:26.864935 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:26.865606 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:26.865614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:26.865703 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:26.866356 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:56:26.866358 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:26.866395 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:56:26.866491 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:26.866837 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:26.866847 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:26.866854 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:56:26.867103 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 02:56:26.867368 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":245,"threads_started":1}
I20260929 02:56:26.869042 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 25430
I20260929 02:56:26.869222 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:56:26.869664 15433 external_mini_cluster.cc:1428] Setting key 590c61261eee46a37425c47718673755
I20260929 02:56:26.869824 25557 heartbeater.cc:347] Connected to a master server at 127.15.18.126:36441
I20260929 02:56:26.870110 25557 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:26.870741 25557 heartbeater.cc:511] Master 127.15.18.126:36441 requested a full tablet report, sending...
I20260929 02:56:26.872195 25240 ts_manager.cc:194] Registered new tserver with Master: 5398afd204e648e9b7b992a88818f6e6 (127.15.18.66:45981)
I20260929 02:56:26.872406 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:36441
--builtin_ntp_servers=127.15.18.84:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:56:27.198024 25560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:27.198316 25560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:27.198446 25560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:27.205786 25560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:27.205911 25560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:27.206063 25560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:56:27.215394 25560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:36441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:27.216801 25560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:27.218601 25560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:27.231655 25565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:27.231684 25568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:27.231609 25566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:27.232468 25560 server_base.cc:1034] running on GCE node
I20260929 02:56:27.233100 25560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:27.234076 25560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:27.235298 25560 hybrid_clock.cc:648] HybridClock initialized: now 1790650587235246 us; error 45 us; skew 500 ppm
I20260929 02:56:27.235713 25560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:56:27.238082 25560 webserver.cc:460] Webserver started at http://127.15.18.67:35107/ using document root <none> and password file <none>
I20260929 02:56:27.238799 25560 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:27.238893 25560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:27.239212 25560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:27.242161 25560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "69c337666bfc4aed80854c3928048b7f"
format_stamp: "Formatted at 2026-09-29 02:56:27 on dist-test-slave-q5f0"
server_key: "eceff83e62ac2c072e5c6552f0fb9f3c"
server_key_iv: "cb84e61de763520772c1c01f4700689d"
server_key_version: "encryptionkey@0"
I20260929 02:56:27.242978 25560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "69c337666bfc4aed80854c3928048b7f"
format_stamp: "Formatted at 2026-09-29 02:56:27 on dist-test-slave-q5f0"
server_key: "eceff83e62ac2c072e5c6552f0fb9f3c"
server_key_iv: "cb84e61de763520772c1c01f4700689d"
server_key_version: "encryptionkey@0"
I20260929 02:56:27.248178 25560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 02:56:27.251991 25574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:27.253463 25560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:56:27.253616 25560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "69c337666bfc4aed80854c3928048b7f"
format_stamp: "Formatted at 2026-09-29 02:56:27 on dist-test-slave-q5f0"
server_key: "eceff83e62ac2c072e5c6552f0fb9f3c"
server_key_iv: "cb84e61de763520772c1c01f4700689d"
server_key_version: "encryptionkey@0"
I20260929 02:56:27.253836 25560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:27.259590 25560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:27.260404 25560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:27.260670 25560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:27.261569 25560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:27.263013 25560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:27.263109 25560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:27.263217 25560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:27.263286 25560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:27.292677 25560 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:35709
I20260929 02:56:27.292724 25686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:35709 every 8 connection(s)
I20260929 02:56:27.294014 25560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:56:27.302457 25428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:27.296686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:36799 (local address 127.15.18.126:36441)
0929 02:56:27.296881 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:27.296885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:27.298556 (+ 1671us) server_negotiation.cc:407] Connection header received
0929 02:56:27.300354 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:27.300359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:27.300426 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:27.300508 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:27.301777 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:27.301784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:27.301788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:27.301807 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:27.301836 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:27.301847 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:27.301964 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:27.302184 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 02:56:27.302241 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:56:27.303366 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:27.297404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:36441 (local address 127.15.18.67:36799)
0929 02:56:27.298398 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:27.298442 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:27.300040 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:27.300745 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:27.300754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:27.300849 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:27.301485 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:56:27.301487 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:27.301524 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:56:27.301612 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:27.301942 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:27.301954 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:27.301963 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:56:27.302195 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 02:56:27.302406 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":288,"threads_started":1}
I20260929 02:56:27.303642 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 25560
I20260929 02:56:27.303817 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:56:27.304313 15433 external_mini_cluster.cc:1428] Setting key c6c5d2144886062d04764f78dad1b516
I20260929 02:56:27.304917 25687 heartbeater.cc:347] Connected to a master server at 127.15.18.126:36441
I20260929 02:56:27.305248 25687 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:27.306062 25687 heartbeater.cc:511] Master 127.15.18.126:36441 requested a full tablet report, sending...
I20260929 02:56:27.307554 25240 ts_manager.cc:194] Registered new tserver with Master: 69c337666bfc4aed80854c3928048b7f (127.15.18.67:35709)
I20260929 02:56:27.319994 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:56:27.326041 25428 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:56:27.324336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.15.18.126:36441)
0929 02:56:27.324562 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:27.324569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:27.325021 (+ 452us) server_negotiation.cc:407] Connection header received
0929 02:56:27.325235 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:27.325240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:27.325297 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:27.325388 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:27.325758 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48758: BlockingRecv error: recv got EOF from 127.0.0.1:48758 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:56:27.326911 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 25299
I20260929 02:56:27.330192 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:27.327861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.15.18.65:45705)
0929 02:56:27.328361 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:27.328370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:27.328396 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:56:27.328452 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:27.328457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:27.328673 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:27.328796 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:27.329599 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:27.329605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:27.329617 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:27.329634 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:27.329675 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:27.329688 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:27.329789 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:27.329867 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 02:56:27.329947 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":240,"threads_started":1}
I20260929 02:56:27.331700 25421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:27.413129 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 25299
I20260929 02:56:27.426234 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 25430
I20260929 02:56:27.439698 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:27.431874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.15.18.66:45981)
0929 02:56:27.433103 (+ 1229us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:27.433110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:27.433139 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:56:27.433201 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:27.433209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:27.433512 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:27.433695 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:27.438883 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:27.438893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:27.438908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:27.438936 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:27.438995 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:27.439023 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:27.439158 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:27.439241 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 02:56:27.439348 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":942,"threads_started":1}
I20260929 02:56:27.441031 25552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:27.521543 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 25430
I20260929 02:56:27.533883 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 25560
I20260929 02:56:27.537959 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:27.534968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.15.18.67:35709)
0929 02:56:27.535637 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:27.535643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:27.535675 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:56:27.535738 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:27.535745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:27.535968 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:27.536112 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:27.537215 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:27.537221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:27.537233 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:56:27.537249 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:27.537288 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:56:27.537302 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:27.537451 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:27.537552 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 02:56:27.537664 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":323,"threads_started":1}
I20260929 02:56:27.539142 25682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:27.621107 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 25560
I20260929 02:56:27.633383 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 25209
I20260929 02:56:27.634514 25270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:27.734678 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 25209
2026-09-29T02:56:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2277 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:56:27 dist-test-slave-q5f0 krb5kdc[25705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:56:27 dist-test-slave-q5f0 krb5kdc[25705](info): set up 2 sockets
Sep 29 02:56:27 dist-test-slave-q5f0 krb5kdc[25705](info): commencing operation
krb5kdc: starting...
W20260929 02:56:28.760526 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:56:28 dist-test-slave-q5f0 krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:56:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:56:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:56:28.985113 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:35503
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:35503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:56:29.305783 25718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:29.306079 25718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:29.306207 25718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:29.313452 25718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:56:29.313570 25718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:29.313633 25718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:29.313714 25718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:56:29.313784 25718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:56:29.324098 25718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:35503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:35503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:29.325443 25718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:29.326727 25718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:29.338658 25726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:29.338658 25724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:29.338727 25723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:29.339392 25718 server_base.cc:1034] running on GCE node
I20260929 02:56:29.340337 25718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:29.341948 25718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:29.343200 25718 hybrid_clock.cc:648] HybridClock initialized: now 1790650589343137 us; error 57 us; skew 500 ppm
I20260929 02:56:29.343638 25718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:29 dist-test-slave-q5f0 krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650589, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:29.346817 25718 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:56:29.348402 25718 webserver.cc:460] Webserver started at http://127.15.18.126:40505/ using document root <none> and password file <none>
I20260929 02:56:29.349083 25718 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:29.349203 25718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:29.349512 25718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:29.352564 25718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "155615d816be4d779ef6094c655495a3"
format_stamp: "Formatted at 2026-09-29 02:56:29 on dist-test-slave-q5f0"
server_key: "ad73921fba37de6432f7a29c962c4db4"
server_key_iv: "5ce6542da4910aa14cdf6479832246e0"
server_key_version: "encryptionkey@0"
I20260929 02:56:29.353317 25718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "155615d816be4d779ef6094c655495a3"
format_stamp: "Formatted at 2026-09-29 02:56:29 on dist-test-slave-q5f0"
server_key: "ad73921fba37de6432f7a29c962c4db4"
server_key_iv: "5ce6542da4910aa14cdf6479832246e0"
server_key_version: "encryptionkey@0"
I20260929 02:56:29.358556 25718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:56:29.362056 25733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:29.363569 25718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:56:29.363746 25718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "155615d816be4d779ef6094c655495a3"
format_stamp: "Formatted at 2026-09-29 02:56:29 on dist-test-slave-q5f0"
server_key: "ad73921fba37de6432f7a29c962c4db4"
server_key_iv: "5ce6542da4910aa14cdf6479832246e0"
server_key_version: "encryptionkey@0"
I20260929 02:56:29.363935 25718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:29.381057 25718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:29.385247 25718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:29.385511 25718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:29.402704 25718 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:35503
I20260929 02:56:29.402740 25784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:35503 every 8 connection(s)
I20260929 02:56:29.404162 25718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:56:29.406378 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 25718
I20260929 02:56:29.406612 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:56:29.407387 15433 external_mini_cluster.cc:1428] Setting key 8759b835901df44e18dd88b6bc06679e
I20260929 02:56:29.408434 25785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:56:29 dist-test-slave-q5f0 krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:56:29.423645 25785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3: Bootstrap starting.
I20260929 02:56:29.427706 25785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3: Neither blocks nor log segments found. Creating new log.
I20260929 02:56:29.428978 25785 log.cc:824] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:56:29.431936 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:29.409383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.15.18.126:35503)
0929 02:56:29.410120 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:29.410132 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:29.410171 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:56:29.411697 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:29.411720 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:29.412116 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:29.412498 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:29.414278 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:29.415669 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:29.417437 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:29.418163 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:29.418350 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:29.422403 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:29.422412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:29.422428 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:29.422468 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:29.426683 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:29.427574 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:29.427583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:29.427591 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:29.427808 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:29.428267 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:29.428272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:29.428275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:29.429032 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:29.429366 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:29.430605 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0929 02:56:29.430924 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":336,"threads_started":1}
I20260929 02:56:29.432170 25785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3: No bootstrap required, opened a new log
I20260929 02:56:29.439431 25785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155615d816be4d779ef6094c655495a3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 35503 } }
I20260929 02:56:29.439888 25785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:56:29.439991 25785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155615d816be4d779ef6094c655495a3, State: Initialized, Role: FOLLOWER
I20260929 02:56:29.440577 25785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155615d816be4d779ef6094c655495a3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 35503 } }
I20260929 02:56:29.440795 25785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:56:29.440923 25785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:56:29.441087 25785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:56:29.443246 25785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155615d816be4d779ef6094c655495a3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 35503 } }
I20260929 02:56:29.443771 25785 leader_election.cc:302] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 155615d816be4d779ef6094c655495a3; no voters:
I20260929 02:56:29.444330 25785 leader_election.cc:288] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:56:29.444449 25790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:56:29.445065 25790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [term 1 LEADER]: Becoming Leader. State: Replica: 155615d816be4d779ef6094c655495a3, State: Running, Role: LEADER
I20260929 02:56:29.445577 25790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155615d816be4d779ef6094c655495a3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 35503 } }
I20260929 02:56:29.446218 25785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:56:29.448249 25792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 155615d816be4d779ef6094c655495a3. Latest consensus state: current_term: 1 leader_uuid: "155615d816be4d779ef6094c655495a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155615d816be4d779ef6094c655495a3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 35503 } } }
I20260929 02:56:29.448562 25792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:56:29.449311 25791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "155615d816be4d779ef6094c655495a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155615d816be4d779ef6094c655495a3" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 35503 } } }
I20260929 02:56:29.449647 25791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:56:29.450232 25797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:56:29.455435 25797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:56:29.464028 25797 catalog_manager.cc:1347] Generated new cluster ID: 078ab43566bb4279883e32827c843d0b
I20260929 02:56:29.464145 25797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:56:29.483246 25797 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:56:29.484802 25797 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:56:29.497583 25797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3: Generated new TSK 0
I20260929 02:56:29.498430 25797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:56:29.562621 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:35503
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:56:29.921685 25811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:29.921989 25811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:29.922137 25811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:29.929370 25811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:29.929494 25811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:29.929644 25811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:56:29.939617 25811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:35503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:29.940937 25811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:29.942268 25811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:29.958030 25817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:29.957949 25816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:29.958042 25819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:29.959138 25811 server_base.cc:1034] running on GCE node
I20260929 02:56:29.959846 25811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:29.960832 25811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:29.962049 25811 hybrid_clock.cc:648] HybridClock initialized: now 1790650589961987 us; error 57 us; skew 500 ppm
I20260929 02:56:29.962368 25811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:29 dist-test-slave-q5f0 krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650589, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:29.965515 25811 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:56:29.967047 25811 webserver.cc:460] Webserver started at http://127.15.18.65:34691/ using document root <none> and password file <none>
I20260929 02:56:29.967804 25811 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:29.967921 25811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:29.968199 25811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:29.971169 25811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "468e317c2ecb4cdbb48b060c055a421b"
format_stamp: "Formatted at 2026-09-29 02:56:29 on dist-test-slave-q5f0"
server_key: "09aff6a431f2181e770b83cf1123dfd9"
server_key_iv: "1562f929469640bdcff198259918b23b"
server_key_version: "encryptionkey@0"
I20260929 02:56:29.972071 25811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "468e317c2ecb4cdbb48b060c055a421b"
format_stamp: "Formatted at 2026-09-29 02:56:29 on dist-test-slave-q5f0"
server_key: "09aff6a431f2181e770b83cf1123dfd9"
server_key_iv: "1562f929469640bdcff198259918b23b"
server_key_version: "encryptionkey@0"
I20260929 02:56:29.977401 25811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:56:29.980849 25826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:29.982209 25811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:56:29.982349 25811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "468e317c2ecb4cdbb48b060c055a421b"
format_stamp: "Formatted at 2026-09-29 02:56:29 on dist-test-slave-q5f0"
server_key: "09aff6a431f2181e770b83cf1123dfd9"
server_key_iv: "1562f929469640bdcff198259918b23b"
server_key_version: "encryptionkey@0"
I20260929 02:56:29.982528 25811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:30.019711 25811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:30.023974 25811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:30.024263 25811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:30.025241 25811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:30.026579 25811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:30.026661 25811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:30.026778 25811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:30.026829 25811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:30.057361 25811 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:44759
I20260929 02:56:30.057422 25938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:44759 every 8 connection(s)
I20260929 02:56:30.058612 25811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:56:30.068827 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 25811
I20260929 02:56:30.069037 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:56:30.069806 15433 external_mini_cluster.cc:1428] Setting key 2385dc8e1bd832345d21a9e53b09f5f3
Sep 29 02:56:30 dist-test-slave-q5f0 krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650589, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:56:30.079540 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:30.061102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:59537 (local address 127.15.18.126:35503)
0929 02:56:30.061545 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:30.061551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:30.062441 (+ 890us) server_negotiation.cc:407] Connection header received
0929 02:56:30.064212 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:30.064217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:30.064286 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:30.064384 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:30.066019 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:30.066761 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:30.068693 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:30.069137 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:30.069255 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:30.073338 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:30.073344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:30.073347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:30.073383 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:30.076034 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:30.076943 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:30.076947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:30.076950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:30.077074 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:30.077538 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:30.077542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:30.077544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:30.077850 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:30.077961 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:30.078874 (+ 913us) server_negotiation.cc:299] Negotiation successful
0929 02:56:30.079141 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":207,"threads_started":1}
I20260929 02:56:30.080278 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:30.061632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:35503 (local address 127.15.18.65:59537)
0929 02:56:30.062291 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:30.062329 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:30.063943 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:30.064580 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:30.064588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:30.065093 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:30.065815 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:30.065838 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:30.066949 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:30.066953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:30.068470 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:30.068485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:30.069403 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:30.069411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:30.069727 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:30.070479 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:30.070503 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:30.073043 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:30.076181 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:30.076191 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:30.076212 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:30.076810 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:30.077170 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:30.077174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:30.077177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:30.077420 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:30.077963 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:30.077971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:30.078436 (+ 465us) client_negotiation.cc:769] Sending connection context
0929 02:56:30.078685 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 02:56:30.079106 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":255,"threads_started":1}
I20260929 02:56:30.081847 25939 heartbeater.cc:347] Connected to a master server at 127.15.18.126:35503
I20260929 02:56:30.082157 25939 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:30.082906 25939 heartbeater.cc:511] Master 127.15.18.126:35503 requested a full tablet report, sending...
I20260929 02:56:30.085608 25750 ts_manager.cc:194] Registered new tserver with Master: 468e317c2ecb4cdbb48b060c055a421b (127.15.18.65:44759)
I20260929 02:56:30.087316 25750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:59537
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:56:30.120510 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:35503
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:56:30.447882 25945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:30.448165 25945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:30.448302 25945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:30.455682 25945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:30.455842 25945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:30.455989 25945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:56:30.465889 25945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:35503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:30.467206 25945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:30.468580 25945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:30.481556 25950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:30.482023 25951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:30.482201 25953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:30.482681 25945 server_base.cc:1034] running on GCE node
I20260929 02:56:30.483249 25945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:30.484329 25945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:30.485522 25945 hybrid_clock.cc:648] HybridClock initialized: now 1790650590485445 us; error 70 us; skew 500 ppm
I20260929 02:56:30.485864 25945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:30 dist-test-slave-q5f0 krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650590, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:30.488972 25945 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:56:30.490334 25945 webserver.cc:460] Webserver started at http://127.15.18.66:42079/ using document root <none> and password file <none>
I20260929 02:56:30.490968 25945 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:30.491062 25945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:30.491307 25945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:30.494129 25945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "83dfaf8dd36d456196ced442930e3131"
format_stamp: "Formatted at 2026-09-29 02:56:30 on dist-test-slave-q5f0"
server_key: "da62433ac0b82b0aa70186521b14c129"
server_key_iv: "b6e41766f878afd0adfacde270f8911a"
server_key_version: "encryptionkey@0"
I20260929 02:56:30.494808 25945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "83dfaf8dd36d456196ced442930e3131"
format_stamp: "Formatted at 2026-09-29 02:56:30 on dist-test-slave-q5f0"
server_key: "da62433ac0b82b0aa70186521b14c129"
server_key_iv: "b6e41766f878afd0adfacde270f8911a"
server_key_version: "encryptionkey@0"
I20260929 02:56:30.499816 25945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 02:56:30.503171 25960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:30.504726 25945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:56:30.504868 25945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "83dfaf8dd36d456196ced442930e3131"
format_stamp: "Formatted at 2026-09-29 02:56:30 on dist-test-slave-q5f0"
server_key: "da62433ac0b82b0aa70186521b14c129"
server_key_iv: "b6e41766f878afd0adfacde270f8911a"
server_key_version: "encryptionkey@0"
I20260929 02:56:30.505036 25945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:30.523964 25945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:30.528280 25945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:30.528558 25945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:30.529523 25945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:30.530890 25945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:30.530970 25945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:30.531093 25945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:30.531144 25945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:30.560310 25945 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:43133
I20260929 02:56:30.560323 26072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:43133 every 8 connection(s)
I20260929 02:56:30.561542 25945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:56:30.561800 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 25945
I20260929 02:56:30.561972 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:56:30.562464 15433 external_mini_cluster.cc:1428] Setting key f0486910ea9201208d2bac78313eeb03
Sep 29 02:56:30 dist-test-slave-q5f0 krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650590, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:56:30.581176 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:30.564107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:47637 (local address 127.15.18.126:35503)
0929 02:56:30.564265 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:30.564269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:30.565748 (+ 1479us) server_negotiation.cc:407] Connection header received
0929 02:56:30.567549 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:30.567554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:30.567637 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:30.567722 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:30.569250 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:30.569921 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:30.571575 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:30.572009 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:30.572109 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:30.575889 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:30.575894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:30.575897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:30.575929 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:30.578252 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:30.578941 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:30.578946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:30.578949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:30.579067 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:30.579471 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:30.579476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:30.579479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:30.579787 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:30.579929 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:30.580615 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 02:56:30.580846 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:56:30.582267 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:30.564809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:35503 (local address 127.15.18.66:47637)
0929 02:56:30.565558 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:30.565613 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:30.567270 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:30.567948 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:30.567957 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:30.568484 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:30.569081 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:30.569099 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:30.570058 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:30.570062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:30.571411 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:30.571425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:30.572221 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:30.572228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:30.572517 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:30.573241 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:30.573273 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:30.575658 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:30.578386 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:30.578395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:30.578415 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:30.578829 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:30.579165 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:30.579171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:30.579174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:30.579368 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:30.579906 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:30.579914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:30.580270 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 02:56:30.580518 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 02:56:30.581069 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":295,"threads_started":1}
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
I20260929 02:56:30.583742 26073 heartbeater.cc:347] Connected to a master server at 127.15.18.126:35503
I20260929 02:56:30.584033 26073 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:30.584761 26073 heartbeater.cc:511] Master 127.15.18.126:35503 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
I20260929 02:56:30.586287 25750 ts_manager.cc:194] Registered new tserver with Master: 83dfaf8dd36d456196ced442930e3131 (127.15.18.66:43133)
I20260929 02:56:30.587145 25750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:47637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:56:30.614984 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:35503
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:56:30.934567 26078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:56:30.934880 26078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:56:30.934996 26078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:56:30.942289 26078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:56:30.942406 26078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:56:30.942550 26078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:56:30.952579 26078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:35503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:56:30.953847 26078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:56:30.955133 26078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:56:30.967747 26084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:30.967788 26083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:56:30.967901 26086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:56:30.968538 26078 server_base.cc:1034] running on GCE node
I20260929 02:56:30.969161 26078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:56:30.970079 26078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:56:30.971300 26078 hybrid_clock.cc:648] HybridClock initialized: now 1790650590971249 us; error 33 us; skew 500 ppm
I20260929 02:56:30.971669 26078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:56:30 dist-test-slave-q5f0 krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650590, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:56:30.974969 26078 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:56:30.976511 26078 webserver.cc:460] Webserver started at http://127.15.18.67:45269/ using document root <none> and password file <none>
I20260929 02:56:30.977238 26078 fs_manager.cc:359] Metadata directory not provided
I20260929 02:56:30.977348 26078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:56:30.977671 26078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:56:30.980783 26078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "e5f42d0f1c644212aacb057824606345"
format_stamp: "Formatted at 2026-09-29 02:56:30 on dist-test-slave-q5f0"
server_key: "2864b03908f4d9544ec0c6d3ca48c383"
server_key_iv: "d630a0289e0fef12e331b8bc02208877"
server_key_version: "encryptionkey@0"
I20260929 02:56:30.981559 26078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "e5f42d0f1c644212aacb057824606345"
format_stamp: "Formatted at 2026-09-29 02:56:30 on dist-test-slave-q5f0"
server_key: "2864b03908f4d9544ec0c6d3ca48c383"
server_key_iv: "d630a0289e0fef12e331b8bc02208877"
server_key_version: "encryptionkey@0"
I20260929 02:56:30.986966 26078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:56:30.990546 26093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:30.992183 26078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:56:30.992336 26078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "e5f42d0f1c644212aacb057824606345"
format_stamp: "Formatted at 2026-09-29 02:56:30 on dist-test-slave-q5f0"
server_key: "2864b03908f4d9544ec0c6d3ca48c383"
server_key_iv: "d630a0289e0fef12e331b8bc02208877"
server_key_version: "encryptionkey@0"
I20260929 02:56:30.992542 26078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:56:31.015106 26078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:56:31.019361 26078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:56:31.019682 26078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:56:31.020697 26078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:56:31.022063 26078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:56:31.022136 26078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:31.022233 26078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:56:31.022289 26078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:56:31.052744 26078 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:43071
I20260929 02:56:31.052791 26205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:43071 every 8 connection(s)
I20260929 02:56:31.053974 26078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:56:31.057636 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 26078
I20260929 02:56:31.057834 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:56:31.058406 15433 external_mini_cluster.cc:1428] Setting key 024e9a1322def37e64eaecf9e062e9a9
Sep 29 02:56:31 dist-test-slave-q5f0 krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650590, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:56:31.075002 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.056524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:41793 (local address 127.15.18.126:35503)
0929 02:56:31.056703 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.056708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.058108 (+ 1400us) server_negotiation.cc:407] Connection header received
0929 02:56:31.060471 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.060476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.060569 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.060644 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:31.062494 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.063281 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.064896 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.065293 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.065392 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.068969 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:31.068974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:31.068977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:31.069013 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:31.071627 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:31.072479 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:31.072484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:31.072488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:31.072660 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:31.073172 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:31.073177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:31.073181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:31.073511 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:31.073641 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.074538 (+ 897us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.074722 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:56:31.076066 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.057055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:35503 (local address 127.15.18.67:41793)
0929 02:56:31.057949 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.057996 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:31.060175 (+ 2179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:31.061155 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:31.061167 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:31.061777 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:56:31.062328 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.062349 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.063439 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.063443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.064740 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.064752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.065500 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.065505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.065756 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.066425 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:56:31.066449 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:56:31.068739 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:56:31.071783 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:31.071791 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:31.071805 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:31.072338 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:31.072774 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:56:31.072779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:56:31.072783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:56:31.073025 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:56:31.073663 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:56:31.073672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:56:31.074161 (+ 489us) client_negotiation.cc:769] Sending connection context
0929 02:56:31.074423 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 02:56:31.074841 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":349,"threads_started":1}
I20260929 02:56:31.077498 26206 heartbeater.cc:347] Connected to a master server at 127.15.18.126:35503
I20260929 02:56:31.077842 26206 heartbeater.cc:464] Registering TS with master...
I20260929 02:56:31.078590 26206 heartbeater.cc:511] Master 127.15.18.126:35503 requested a full tablet report, sending...
I20260929 02:56:31.080080 25750 ts_manager.cc:194] Registered new tserver with Master: e5f42d0f1c644212aacb057824606345 (127.15.18.67:43071)
I20260929 02:56:31.080909 25750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:41793
I20260929 02:56:31.086473 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:56:31.090620 25939 heartbeater.cc:503] Master 127.15.18.126:35503 was elected leader, sending a full tablet report...
I20260929 02:56:31.102679 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.090865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.15.18.126:35503)
0929 02:56:31.091076 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.091084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.091483 (+ 399us) server_negotiation.cc:407] Connection header received
0929 02:56:31.091669 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.091674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.091787 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.091869 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:31.093192 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.093904 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.095083 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.095488 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.095637 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.097753 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:31.097758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:31.097761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:31.097791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:31.100225 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:31.100815 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:31.100820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:31.100823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:31.100958 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:31.101315 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:31.101320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:31.101322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:31.101638 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:31.101861 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.102169 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.102369 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 02:56:31.106956 25750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:56:31.113864 25750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:56:31.135807 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.128223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.15.18.67:43071)
0929 02:56:31.129141 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.129146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.129169 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:56:31.129255 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.129262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.129463 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.129639 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:56:31.130409 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.131798 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.133823 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.134717 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.134828 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.135061 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.135309 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.135465 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":657,"threads_started":1}
I20260929 02:56:31.135877 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.125769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44759 (local address 127.0.0.1:50262)
0929 02:56:31.126540 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.126585 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:31.126768 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:31.128762 (+ 1994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:31.128766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:31.128783 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:56:31.129121 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.129131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.130840 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.130845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.133158 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.133172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.135061 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.135068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.135234 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.135265 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:56:31.135331 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 02:56:31.135456 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":66,"thread_start_us":1195,"threads_started":1}
I20260929 02:56:31.136329 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.126002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.15.18.65:44759)
0929 02:56:31.128211 (+ 2209us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.128219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.128245 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:56:31.128349 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.128356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.128570 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.128865 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:56:31.129304 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.130725 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.133334 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.134828 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.135594 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.135802 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.135865 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.135998 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2069,"thread_start_us":1935,"threads_started":1}
I20260929 02:56:31.136382 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.126420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.15.18.66:43133)
0929 02:56:31.127933 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.127941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.127968 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:56:31.128074 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.128082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.128339 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.128523 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:56:31.129799 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.131213 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.133590 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.134891 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.134988 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.135170 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.135721 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.135886 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1366,"thread_start_us":1268,"threads_started":1}
I20260929 02:56:31.136595 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.127776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:43071 (local address 127.0.0.1:46492)
0929 02:56:31.128462 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.128482 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:31.128575 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:31.129879 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:31.129883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:31.129907 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:56:31.130235 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.130246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.131927 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.131932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.133683 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.133696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.134881 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.134887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.135085 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.135139 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 02:56:31.135257 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 02:56:31.135454 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":4,"thread_start_us":585,"threads_started":1}
I20260929 02:56:31.137343 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.125864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:43133 (local address 127.0.0.1:43824)
0929 02:56:31.127582 (+ 1718us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.127604 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:31.127721 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:31.128726 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:31.128733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:31.128761 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:56:31.129611 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.129623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.131348 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.131352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.133418 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.133430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.135347 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.135354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.135527 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.135567 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 02:56:31.136859 (+ 1292us) client_negotiation.cc:240] Negotiation successful
0929 02:56:31.136985 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"mutex_wait_us":1,"thread_start_us":1408,"threads_started":1}
I20260929 02:56:31.140501 26008 tablet_service.cc:1467] Processing CreateTablet for tablet 5f751f75d8014bf7ad6d9ec5218e744b (DEFAULT_TABLE table=test-table [id=f5d972f987154b98872c58eafe9f8a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:56:31.140487 25874 tablet_service.cc:1467] Processing CreateTablet for tablet 5f751f75d8014bf7ad6d9ec5218e744b (DEFAULT_TABLE table=test-table [id=f5d972f987154b98872c58eafe9f8a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:56:31.140487 26141 tablet_service.cc:1467] Processing CreateTablet for tablet 5f751f75d8014bf7ad6d9ec5218e744b (DEFAULT_TABLE table=test-table [id=f5d972f987154b98872c58eafe9f8a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:56:31.141741 26008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f751f75d8014bf7ad6d9ec5218e744b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:56:31.141827 25874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f751f75d8014bf7ad6d9ec5218e744b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:56:31.141824 26141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f751f75d8014bf7ad6d9ec5218e744b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:56:31.153740 26224 tablet_bootstrap.cc:492] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: Bootstrap starting.
I20260929 02:56:31.155936 26225 tablet_bootstrap.cc:492] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b: Bootstrap starting.
I20260929 02:56:31.158286 26224 tablet_bootstrap.cc:654] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: Neither blocks nor log segments found. Creating new log.
I20260929 02:56:31.159608 26224 log.cc:824] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: Log is configured to *not* fsync() on all Append() calls
I20260929 02:56:31.159683 26225 tablet_bootstrap.cc:654] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b: Neither blocks nor log segments found. Creating new log.
I20260929 02:56:31.160261 26226 tablet_bootstrap.cc:492] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: Bootstrap starting.
I20260929 02:56:31.160936 26225 log.cc:824] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:56:31.163424 26226 tablet_bootstrap.cc:654] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: Neither blocks nor log segments found. Creating new log.
I20260929 02:56:31.163609 26224 tablet_bootstrap.cc:492] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: No bootstrap required, opened a new log
I20260929 02:56:31.163868 26224 ts_tablet_manager.cc:1397] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260929 02:56:31.164646 26226 log.cc:824] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: Log is configured to *not* fsync() on all Append() calls
I20260929 02:56:31.165813 26225 tablet_bootstrap.cc:492] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b: No bootstrap required, opened a new log
I20260929 02:56:31.166147 26225 ts_tablet_manager.cc:1397] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 02:56:31.168879 26226 tablet_bootstrap.cc:492] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: No bootstrap required, opened a new log
I20260929 02:56:31.169145 26226 ts_tablet_manager.cc:1397] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:56:31.171675 26224 raft_consensus.cc:348] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 } } peers { permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 } } peers { permanent_uuid: "e5f42d0f1c644212aacb057824606345" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 43071 } }
I20260929 02:56:31.172049 26224 raft_consensus.cc:374] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:56:31.172161 26224 raft_consensus.cc:729] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83dfaf8dd36d456196ced442930e3131, State: Initialized, Role: FOLLOWER
I20260929 02:56:31.172727 26224 consensus_queue.cc:260] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 } } peers { permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 } } peers { permanent_uuid: "e5f42d0f1c644212aacb057824606345" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 43071 } }
I20260929 02:56:31.173840 26073 heartbeater.cc:503] Master 127.15.18.126:35503 was elected leader, sending a full tablet report...
I20260929 02:56:31.174397 26224 ts_tablet_manager.cc:1428] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20260929 02:56:31.175992 26225 raft_consensus.cc:348] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 } } peers { permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 } } peers { permanent_uuid: "e5f42d0f1c644212aacb057824606345" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 43071 } }
I20260929 02:56:31.176436 26225 raft_consensus.cc:374] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:56:31.176530 26225 raft_consensus.cc:729] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468e317c2ecb4cdbb48b060c055a421b, State: Initialized, Role: FOLLOWER
I20260929 02:56:31.177140 26225 consensus_queue.cc:260] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 } } peers { permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 } } peers { permanent_uuid: "e5f42d0f1c644212aacb057824606345" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 43071 } }
I20260929 02:56:31.178970 26225 ts_tablet_manager.cc:1428] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260929 02:56:31.179862 26226 raft_consensus.cc:348] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 } } peers { permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 } } peers { permanent_uuid: "e5f42d0f1c644212aacb057824606345" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 43071 } }
I20260929 02:56:31.180330 26226 raft_consensus.cc:374] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:56:31.180430 26226 raft_consensus.cc:729] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f42d0f1c644212aacb057824606345, State: Initialized, Role: FOLLOWER
I20260929 02:56:31.181123 26226 consensus_queue.cc:260] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 } } peers { permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 } } peers { permanent_uuid: "e5f42d0f1c644212aacb057824606345" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 43071 } }
I20260929 02:56:31.182202 26206 heartbeater.cc:503] Master 127.15.18.126:35503 was elected leader, sending a full tablet report...
I20260929 02:56:31.182965 26226 ts_tablet_manager.cc:1428] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260929 02:56:31.305634 26207 tablet.cc:2369] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:56:31.310951 25940 tablet.cc:2369] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:56:31.313488 26074 tablet.cc:2369] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:56:31.477101 26232 raft_consensus.cc:482] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:56:31.477373 26232 raft_consensus.cc:504] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 } } peers { permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 } } peers { permanent_uuid: "e5f42d0f1c644212aacb057824606345" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 43071 } }
I20260929 02:56:31.479166 26232 leader_election.cc:288] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 468e317c2ecb4cdbb48b060c055a421b (127.15.18.65:44759), 83dfaf8dd36d456196ced442930e3131 (127.15.18.66:43133)
I20260929 02:56:31.484834 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.479736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:51977 (local address 127.15.18.66:43133)
0929 02:56:31.479968 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.479974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.480198 (+ 224us) server_negotiation.cc:407] Connection header received
0929 02:56:31.480406 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.480410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.480472 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.480566 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:56:31.480910 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.481616 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.483182 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.484006 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.484149 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.484222 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.484456 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.484586 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:56:31.484841 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.479696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:43133 (local address 127.15.18.67:51977)
0929 02:56:31.480066 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.480096 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:31.480259 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:31.480596 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:31.480601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:31.480628 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:56:31.480785 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.480793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.481734 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.481740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.483037 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.483048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.484147 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.484153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.484324 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.484347 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:56:31.484424 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:56:31.484564 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260929 02:56:31.485189 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.479782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:49173 (local address 127.15.18.65:44759)
0929 02:56:31.480012 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.480017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.480523 (+ 506us) server_negotiation.cc:407] Connection header received
0929 02:56:31.480657 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.480661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.480719 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.480788 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:56:31.481266 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.481972 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.483614 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.484366 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.484513 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.484581 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.484840 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.484963 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 02:56:31.485249 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.479733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44759 (local address 127.15.18.67:49173)
0929 02:56:31.480429 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.480441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 02:56:31.480539 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:56:31.480819 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:56:31.480823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:56:31.480844 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:56:31.481124 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.481132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.482115 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.482121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.483446 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:56:31.483457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.484555 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.484561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:56:31.484702 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.484723 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 02:56:31.484800 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:56:31.484935 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":394,"threads_started":1}
I20260929 02:56:31.485975 26028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f751f75d8014bf7ad6d9ec5218e744b" candidate_uuid: "e5f42d0f1c644212aacb057824606345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83dfaf8dd36d456196ced442930e3131" is_pre_election: true
I20260929 02:56:31.486546 26028 raft_consensus.cc:2454] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5f42d0f1c644212aacb057824606345 in term 0.
I20260929 02:56:31.486445 25894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f751f75d8014bf7ad6d9ec5218e744b" candidate_uuid: "e5f42d0f1c644212aacb057824606345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468e317c2ecb4cdbb48b060c055a421b" is_pre_election: true
I20260929 02:56:31.486963 25894 raft_consensus.cc:2454] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5f42d0f1c644212aacb057824606345 in term 0.
I20260929 02:56:31.487255 26096 leader_election.cc:302] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83dfaf8dd36d456196ced442930e3131, e5f42d0f1c644212aacb057824606345; no voters:
I20260929 02:56:31.487663 26232 raft_consensus.cc:2781] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:56:31.487790 26232 raft_consensus.cc:482] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:56:31.487890 26232 raft_consensus.cc:3037] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:56:31.490129 26232 raft_consensus.cc:504] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 } } peers { permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 } } peers { permanent_uuid: "e5f42d0f1c644212aacb057824606345" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 43071 } }
I20260929 02:56:31.490739 26232 leader_election.cc:288] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [CANDIDATE]: Term 1 election: Requested vote from peers 468e317c2ecb4cdbb48b060c055a421b (127.15.18.65:44759), 83dfaf8dd36d456196ced442930e3131 (127.15.18.66:43133)
I20260929 02:56:31.491115 25894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f751f75d8014bf7ad6d9ec5218e744b" candidate_uuid: "e5f42d0f1c644212aacb057824606345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468e317c2ecb4cdbb48b060c055a421b"
I20260929 02:56:31.491225 26028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f751f75d8014bf7ad6d9ec5218e744b" candidate_uuid: "e5f42d0f1c644212aacb057824606345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83dfaf8dd36d456196ced442930e3131"
I20260929 02:56:31.491300 25894 raft_consensus.cc:3037] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:56:31.491426 26028 raft_consensus.cc:3037] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:56:31.493729 25894 raft_consensus.cc:2454] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5f42d0f1c644212aacb057824606345 in term 1.
I20260929 02:56:31.493863 26028 raft_consensus.cc:2454] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5f42d0f1c644212aacb057824606345 in term 1.
I20260929 02:56:31.494124 26095 leader_election.cc:302] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 468e317c2ecb4cdbb48b060c055a421b, e5f42d0f1c644212aacb057824606345; no voters:
I20260929 02:56:31.494411 26232 raft_consensus.cc:2781] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:56:31.494814 26232 raft_consensus.cc:686] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 1 LEADER]: Becoming Leader. State: Replica: e5f42d0f1c644212aacb057824606345, State: Running, Role: LEADER
I20260929 02:56:31.495343 26232 consensus_queue.cc:237] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 } } peers { permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 } } peers { permanent_uuid: "e5f42d0f1c644212aacb057824606345" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 43071 } }
I20260929 02:56:31.500083 25749 catalog_manager.cc:5495] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5f42d0f1c644212aacb057824606345 (127.15.18.67). New cstate: current_term: 1 leader_uuid: "e5f42d0f1c644212aacb057824606345" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5f42d0f1c644212aacb057824606345" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 43071 } health_report { overall_health: HEALTHY } } }
I20260929 02:56:31.593884 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.588356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.15.18.67:43071)
0929 02:56:31.588539 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.588544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.588566 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:56:31.588767 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.588771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.588859 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.589002 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:56:31.589472 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.590250 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.591667 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.592074 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.592206 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.592723 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:56:31.593277 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:56:31.593398 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.593580 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.593665 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:56:31.603286 26028 raft_consensus.cc:1261] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131 [term 1 FOLLOWER]: Refusing update from remote peer e5f42d0f1c644212aacb057824606345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:56:31.603286 25894 raft_consensus.cc:1261] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b [term 1 FOLLOWER]: Refusing update from remote peer e5f42d0f1c644212aacb057824606345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:56:31.604329 26232 consensus_queue.cc:1035] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83dfaf8dd36d456196ced442930e3131" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:56:31.604674 26234 consensus_queue.cc:1035] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "468e317c2ecb4cdbb48b060c055a421b" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:56:31.616780 26237 mvcc.cc:205] Tried to move back new op lower bound from 7334504823194591232 to 7334504822769340416. Current Snapshot: MvccSnapshot[applied={T|T < 7334504823194591232}]
I20260929 02:56:31.632411 25750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:56:31.632697 25750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:56:31.636240 25750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 155615d816be4d779ef6094c655495a3: Sending DeleteTablet for 3 replicas of tablet 5f751f75d8014bf7ad6d9ec5218e744b
I20260929 02:56:31.637669 25874 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f751f75d8014bf7ad6d9ec5218e744b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:56:31 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:50262
I20260929 02:56:31.637790 26141 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f751f75d8014bf7ad6d9ec5218e744b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:56:31 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:46492
I20260929 02:56:31.638432 26008 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f751f75d8014bf7ad6d9ec5218e744b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:56:31 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:43824
I20260929 02:56:31.639442 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 25811
I20260929 02:56:31.663161 26248 tablet_replica.cc:330] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: stopping tablet replica
I20260929 02:56:31.663817 26248 raft_consensus.cc:2229] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:56:31.664351 26248 raft_consensus.cc:2258] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:56:31.665066 26246 tablet_replica.cc:330] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b: stopping tablet replica
I20260929 02:56:31.665485 26246 raft_consensus.cc:2229] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:56:31.665963 26246 raft_consensus.cc:2258] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 02:56:31 dist-test-slave-q5f0 krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:56:31.670359 26246 ts_tablet_manager.cc:1905] T 5f751f75d8014bf7ad6d9ec5218e744b P 468e317c2ecb4cdbb48b060c055a421b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:56:31.675740 26248 ts_tablet_manager.cc:1905] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:56:31.677018 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.643018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.15.18.65:44759)
0929 02:56:31.643206 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.643212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.655405 (+ 12193us) server_negotiation.cc:407] Connection header received
0929 02:56:31.655951 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.655958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.656051 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.656137 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:31.658014 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.659206 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.661235 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.661938 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.662072 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.668672 (+ 6600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:31.668691 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:31.668712 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:31.668784 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:31.673049 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:31.674039 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:31.674051 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:31.674055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:31.674251 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:31.674733 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:31.674743 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:31.674747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:31.675258 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:31.675622 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.676341 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.676613 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:56:31.678881 25934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:31.682803 26248 ts_tablet_manager.cc:1918] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:56:31.682989 26248 log.cc:1196] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/5f751f75d8014bf7ad6d9ec5218e744b
I20260929 02:56:31.683995 26247 tablet_replica.cc:330] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: stopping tablet replica
I20260929 02:56:31.684479 26247 raft_consensus.cc:2229] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:56:31.685195 26247 raft_consensus.cc:2258] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:56:31.683486 26248 ts_tablet_manager.cc:1939] T 5f751f75d8014bf7ad6d9ec5218e744b P 83dfaf8dd36d456196ced442930e3131: Deleting consensus metadata
I20260929 02:56:31.687769 26247 ts_tablet_manager.cc:1905] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:56:31.688019 25736 catalog_manager.cc:4841] TS 83dfaf8dd36d456196ced442930e3131 (127.15.18.66:43133): tablet 5f751f75d8014bf7ad6d9ec5218e744b (table test-table [id=f5d972f987154b98872c58eafe9f8a5e]) successfully deleted
I20260929 02:56:31.694999 26247 ts_tablet_manager.cc:1918] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:56:31.695188 26247 log.cc:1196] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/5f751f75d8014bf7ad6d9ec5218e744b
I20260929 02:56:31.695662 26247 ts_tablet_manager.cc:1939] T 5f751f75d8014bf7ad6d9ec5218e744b P e5f42d0f1c644212aacb057824606345: Deleting consensus metadata
I20260929 02:56:31.697243 25736 catalog_manager.cc:4841] TS e5f42d0f1c644212aacb057824606345 (127.15.18.67:43071): tablet 5f751f75d8014bf7ad6d9ec5218e744b (table test-table [id=f5d972f987154b98872c58eafe9f8a5e]) successfully deleted
I20260929 02:56:31.798332 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 25811
W20260929 02:56:31.818301 25735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.18.65:44759 (error 108)
I20260929 02:56:31.818418 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 25945
I20260929 02:56:31.825443 26252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.819807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44759 (local address 127.0.0.1:50278)
0929 02:56:31.825068 (+ 5261us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.825216 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5123,"thread_start_us":5027,"threads_started":1}
W20260929 02:56:31.825754 25735 catalog_manager.cc:4584] TS 468e317c2ecb4cdbb48b060c055a421b (127.15.18.65:44759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f751f75d8014bf7ad6d9ec5218e744b: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44759: connect: Connection refused (error 111)
Sep 29 02:56:31 dist-test-slave-q5f0 krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:56:31.835878 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.819202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.15.18.66:43133)
0929 02:56:31.819379 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.819383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.819481 (+ 98us) server_negotiation.cc:407] Connection header received
0929 02:56:31.819726 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.819733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.819824 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.819932 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:31.821433 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.822668 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.824715 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.825237 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.825360 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.829600 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:31.829615 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:31.829632 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:31.829674 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:31.832561 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:31.833507 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:31.833515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:31.833518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:31.833714 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:31.834156 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:31.834160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:31.834162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:31.834631 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:31.834875 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.835257 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.835452 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:56:31.836591 26068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:31.879616 26252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.879007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44759 (local address 127.0.0.1:50290)
0929 02:56:31.879270 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.879385 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 02:56:31.880836 26252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.880394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44759 (local address 127.0.0.1:50302)
0929 02:56:31.880573 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.880663 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 02:56:31.935824 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 25945
I20260929 02:56:31.941452 26252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.940924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44759 (local address 127.0.0.1:50312)
0929 02:56:31.941158 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.941274 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 02:56:31.942680 26252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.942216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44759 (local address 127.0.0.1:50328)
0929 02:56:31.942385 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:31.942480 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 02:56:31.949975 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 26078
Sep 29 02:56:31 dist-test-slave-q5f0 krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:56:31.967815 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:31.950836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.15.18.67:43071)
0929 02:56:31.950997 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:56:31.951001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:56:31.951058 (+ 57us) server_negotiation.cc:407] Connection header received
0929 02:56:31.951229 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:56:31.951232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:56:31.951292 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:56:31.951372 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:56:31.953061 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.953929 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.955778 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:56:31.956236 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:56:31.956344 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:56:31.961899 (+ 5555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:56:31.961912 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:56:31.961930 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:56:31.961972 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:56:31.964946 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:31.965784 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:31.965791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:31.965793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:31.965973 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:56:31.966357 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:56:31.966359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:56:31.966361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:56:31.966704 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:56:31.966942 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 02:56:31.967282 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 02:56:31.967442 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:56:31.968545 26201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:32.023557 26252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:32.022936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44759 (local address 127.0.0.1:50334)
0929 02:56:32.023207 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:32.023337 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 02:56:32.024729 26252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:32.024267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44759 (local address 127.0.0.1:50348)
0929 02:56:32.024472 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:32.024558 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 02:56:32.091136 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 26078
I20260929 02:56:32.107003 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 25718
I20260929 02:56:32.108132 25780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:56:32.216384 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 25718
I20260929 02:56:32.216423 26252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:56:32.215362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44759 (local address 127.0.0.1:50352)
0929 02:56:32.215793 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 02:56:32.215940 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278}
2026-09-29T02:56:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4484 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:56:32 dist-test-slave-q5f0 krb5kdc[26258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:56:32 dist-test-slave-q5f0 krb5kdc[26258](info): set up 2 sockets
Sep 29 02:56:32 dist-test-slave-q5f0 krb5kdc[26258](info): commencing operation
krb5kdc: starting...
W20260929 02:56:33.232777 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:56:33 dist-test-slave-q5f0 krb5kdc[26258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:56:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/rangeradmin_127.15.18.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/rangeradmin_127.15.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/rangerlookup_127.15.18.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/rangerlookup_127.15.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/HTTP_127.15.18.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/HTTP_127.15.18.84@KRBTEST.COM.keytab.
I20260929 02:56:33.550714 15433 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxb3wIY/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/postgres -l logfile start
2026-09-29 02:56:36.106 UTC [26288] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 02:56:36.106 UTC [26288] LOG: listening on IPv4 address "127.15.18.84", port 38399
2026-09-29 02:56:36.110 UTC [26288] LOG: listening on Unix socket "/tmp/.s.PGSQL.38399"
2026-09-29 02:56:36.130 UTC [26291] LOG: database system was shut down at 2026-09-29 02:56:34 UTC
2026-09-29 02:56:36.133 UTC [26288] LOG: database system is ready to accept connections
I20260929 02:56:37.024302 15433 mini_postgres.cc:96] Postgres bound to 38399
2026-09-29 02:56:37.040 UTC [26299] FATAL: database "slave" does not exist
127.15.18.84:38399 - accepting connections
I20260929 02:56:37.041127 15433 mini_ranger.cc:161] Starting Ranger...
I20260929 02:56:37.066720 15433 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 02:56:37.599453 15433 mini_ranger.cc:87] Created ranger Postgres database
I20260929 02:56:37.599635 15433 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-admin
2026-09-29 02:56:37,893 [I] DB FLAVOR :POSTGRES
2026-09-29 02:56:37,894 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:56:37,894 [I] Checking connection..
2026-09-29 02:56:37,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 02:56:38,142 [I] Checking connection passed.
2026-09-29 02:56:38,143 [I] --------- Verifying version history table ---------
2026-09-29 02:56:38,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:56:38,404 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:56:38,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:56:38,630 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:56:38,630 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 02:56:38,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 02:56:38.862 UTC [26363] WARNING: there is no transaction in progress
2026-09-29 02:56:38,868 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 02:56:38,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:56:39,109 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 02:56:39,109 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:56:39,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 02:56:39,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 02:56:39,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q5f0.c.gcp-upstream.internal','N') ;"
2026-09-29 02:56:39,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:56:40,107 [I] Table x_portal_user does not exist in database ranger
2026-09-29 02:56:40,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:56:40,353 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 02:56:40,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:56:40,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:56:40,842 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 02:56:40,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 02:56:41.820 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:41.837 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:41.856 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:41.872 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:41.890 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:41.911 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:41.976 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:41.983 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:41.998 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:42.008 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:42.018 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:42.033 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:42.043 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:42.050 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:42.056 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:42.063 UTC [26471] WARNING: there is no transaction in progress
2026-09-29 02:56:42,726 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 02:56:42,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 02:56:42,956 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 02:56:42,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:56:43,193 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 02:56:43,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:56:43,439 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 02:56:43,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:56:43,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:56:43,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q5f0.c.gcp-upstream.internal';"
2026-09-29 02:56:44,187 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 02:56:44,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 02:56:44,437 [I] DB_PATCHES have already been applied
I20260929 02:56:44.443149 15433 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxb3wIY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 02:56:44.443236 15433 mini_ranger.cc:193] Using host: 127.15.18.84
I20260929 02:56:44.457090 15433 mini_ranger.cc:227] Ranger admin URL: http://127.15.18.84:34637
Sep 29, 2026 2:56:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:56:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:56:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:56:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:56:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:56:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:56:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 02:56:45.467165 15433 mini_ranger.cc:160] Time spent starting Ranger: real 8.426s user 0.002s sys 0.037s
Sep 29, 2026 2:57:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:57:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 2:57:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:57:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:57:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 02:57:07.232313 15433 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/rangerkms_127.15.18.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/rangerkms_127.15.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.18.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.18.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.18.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/HTTP_127.15.18.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.18.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/HTTP_127.15.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 02:57:07 dist-test-slave-q5f0 krb5kdc[26258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650627, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 02:57:07.401853 15433 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 02:57:07.422079 15433 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 02:57:08.382094 15433 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 02:57:08.382246 15433 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-kms
I20260929 02:57:08.382354 15433 mini_ranger_kms.cc:227] Using postgres at 127.15.18.84:38399
2026-09-29 02:57:08,777 [I] DB FLAVOR :POSTGRES
2026-09-29 02:57:08,778 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:57:08,778 [I] Checking connection
2026-09-29 02:57:08,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 02:57:09,015 [I] connection success
2026-09-29 02:57:09,016 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 02:57:09,016 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 02:57:09,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 02:57:09,262 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 02:57:09,262 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:57:09,263 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 02:57:09,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:38399/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 02:57:09,529 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 02:57:10.035789 15433 mini_ranger_kms.cc:310] Created kms service
I20260929 02:57:10.233134 15433 mini_ranger_kms.cc:326] Created kudu user
I20260929 02:57:10.278585 15433 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 02:57:10 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650627, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:57:10.685017 15433 mini_ranger_kms.cc:384] Added ranger policy
I20260929 02:57:10.685343 15433 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 02:57:10.685387 15433 mini_ranger_kms.cc:240] Using host: 127.15.18.84
I20260929 02:57:10.705964 15433 mini_ranger_kms.cc:276] Ranger KMS PID: 26651
I20260929 02:57:10.706136 15433 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.18.84:39927
Sep 29, 2026 2:57:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:57:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:57:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:57:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:57:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:57:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 02:57:12.758572 15433 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.357s user 0.112s sys 0.303s
I20260929 02:57:12.758814 15433 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 02:57:12.759024 15433 mini_ranger_kms.cc:401] 127.15.18.84:39927/kms/v1/keys
Sep 29 02:57:12 dist-test-slave-q5f0 krb5kdc[26258](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650632, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.18.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:57:13 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650632, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.18.84@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
Sep 29 02:57:13 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650632, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.18.84@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
Sep 29 02:57:16 dist-test-slave-q5f0 krb5kdc[26258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:57:16.067711 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:39187
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:39187
--ranger_config_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:39927/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:57:16.393903 26702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:57:16.394205 26702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:57:16.394313 26702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:57:16.401557 26702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:57:16.401666 26702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:57:16.401721 26702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:57:16.401796 26702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:57:16.401854 26702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:57:16.412249 26702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.18.84:39927/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:39187
--ranger_config_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:39187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:57:16.413578 26702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:57:16.414877 26702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:57:16.428309 26708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:57:16.428328 26710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:57:16.428815 26702 server_base.cc:1034] running on GCE node
W20260929 02:57:16.429013 26707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:57:16.429855 26702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:57:16.431260 26702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:57:16.432473 26702 hybrid_clock.cc:648] HybridClock initialized: now 1790650636432391 us; error 81 us; skew 500 ppm
I20260929 02:57:16.432816 26702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:57:16 dist-test-slave-q5f0 krb5kdc[26258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:57:16.439327 26702 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:57:16.441097 26702 webserver.cc:460] Webserver started at http://127.15.18.126:33739/ using document root <none> and password file <none>
I20260929 02:57:16.441778 26702 fs_manager.cc:359] Metadata directory not provided
I20260929 02:57:16.441892 26702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:57:16.442190 26702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:57:16 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:57:16.504129 26702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "9a5f27948dd24b57b71d2da2e19ca36d"
format_stamp: "Formatted at 2026-09-29 02:57:16 on dist-test-slave-q5f0"
server_key: "8177e2045fa6d277f7b2cd69b9a874c5"
server_key_iv: "271799d89047f518b2117b6b986a2946"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:16.504918 26702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "9a5f27948dd24b57b71d2da2e19ca36d"
format_stamp: "Formatted at 2026-09-29 02:57:16 on dist-test-slave-q5f0"
server_key: "8177e2045fa6d277f7b2cd69b9a874c5"
server_key_iv: "271799d89047f518b2117b6b986a2946"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:16.510373 26702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:57:16.530843 26717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:57:16.532402 26702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:57:16.532593 26702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "9a5f27948dd24b57b71d2da2e19ca36d"
format_stamp: "Formatted at 2026-09-29 02:57:16 on dist-test-slave-q5f0"
server_key: "8177e2045fa6d277f7b2cd69b9a874c5"
server_key_iv: "271799d89047f518b2117b6b986a2946"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:16.532817 26702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:57:16.552591 26702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:57:16.556602 26702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:57:16.556876 26702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:57:16.574661 26702 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:39187
I20260929 02:57:16.574677 26768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:39187 every 8 connection(s)
I20260929 02:57:16.576007 26702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:57:16.579715 26769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:57:16.582839 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 26702
I20260929 02:57:16.583042 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:57:16.583873 15433 external_mini_cluster.cc:1428] Setting key ab5dc82e758cf85ddd98e74393825eef
I20260929 02:57:16.593731 26769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d: Bootstrap starting.
I20260929 02:57:16.598224 26769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d: Neither blocks nor log segments found. Creating new log.
I20260929 02:57:16.599568 26769 log.cc:824] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:57:16.602689 26769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d: No bootstrap required, opened a new log
Sep 29 02:57:16 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:57:16.611956 26769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f27948dd24b57b71d2da2e19ca36d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39187 } }
I20260929 02:57:16.612363 26769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:57:16.612454 26769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5f27948dd24b57b71d2da2e19ca36d, State: Initialized, Role: FOLLOWER
I20260929 02:57:16.613078 26769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f27948dd24b57b71d2da2e19ca36d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39187 } }
I20260929 02:57:16.613322 26769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:57:16.613451 26769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:57:16.613610 26769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:57:16.615924 26769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f27948dd24b57b71d2da2e19ca36d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39187 } }
I20260929 02:57:16.616426 26769 leader_election.cc:302] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a5f27948dd24b57b71d2da2e19ca36d; no voters:
I20260929 02:57:16.617040 26769 leader_election.cc:288] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:57:16.617146 26774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:57:16.618314 26774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [term 1 LEADER]: Becoming Leader. State: Replica: 9a5f27948dd24b57b71d2da2e19ca36d, State: Running, Role: LEADER
I20260929 02:57:16.619006 26774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f27948dd24b57b71d2da2e19ca36d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39187 } }
I20260929 02:57:16.619403 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:16.586150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.15.18.126:39187)
0929 02:57:16.597773 (+ 11623us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:16.597789 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:16.598613 (+ 824us) server_negotiation.cc:407] Connection header received
0929 02:57:16.601003 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:16.601030 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:16.601448 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:16.601783 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:57:16.603392 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:16.604872 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:16.606687 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:16.607434 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:16.607626 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:16.611973 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:57:16.611982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:57:16.611998 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:57:16.612037 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:57:16.614927 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:16.615681 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:16.615687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:16.615695 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:16.615853 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:16.616229 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:16.616233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:16.616235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:16.616875 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:57:16.617160 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:16.617690 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 02:57:16.618116 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11365,"thread_start_us":11557,"threads_started":1}
I20260929 02:57:16.623452 26774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a5f27948dd24b57b71d2da2e19ca36d. Latest consensus state: current_term: 1 leader_uuid: "9a5f27948dd24b57b71d2da2e19ca36d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f27948dd24b57b71d2da2e19ca36d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39187 } } }
I20260929 02:57:16.623873 26774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [sys.catalog]: This master's current role is: LEADER
I20260929 02:57:16.624188 26776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a5f27948dd24b57b71d2da2e19ca36d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f27948dd24b57b71d2da2e19ca36d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39187 } } }
I20260929 02:57:16.624557 26776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [sys.catalog]: This master's current role is: LEADER
I20260929 02:57:16.625523 26769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:57:16.630642 26778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:57:16.635581 26778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:57:16.636376 26769 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:57:16.663290 26778 catalog_manager.cc:1347] Generated new cluster ID: 4831795693c441c696c070a1789bed6c
I20260929 02:57:16.663537 26778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:57:16.694216 26778 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:57:16.696273 26778 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:57:16.708763 26778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d: Generated new TSK 0
I20260929 02:57:16.709906 26778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:57:17.438707 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:39927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39187
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:57:17 dist-test-slave-q5f0 krb5kdc[26258](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 02:57:17.874521 26814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:57:17.874941 26814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:57:17.875098 26814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:57:17.886569 26814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:57:17.886721 26814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:57:17.886919 26814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:57:17.902484 26814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.18.84:39927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:57:17.904309 26814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:57:17.906072 26814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:57:17.924326 26824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:57:17.925868 26825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:57:17.941468 26827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:57:17.941829 26814 server_base.cc:1034] running on GCE node
I20260929 02:57:17.942546 26814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:57:17.943686 26814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:57:17.944922 26814 hybrid_clock.cc:648] HybridClock initialized: now 1790650637944865 us; error 55 us; skew 500 ppm
I20260929 02:57:17.945261 26814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:57:17 dist-test-slave-q5f0 krb5kdc[26258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:57:17.952243 26814 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:57:17.955878 26814 webserver.cc:460] Webserver started at http://127.15.18.65:40731/ using document root <none> and password file <none>
I20260929 02:57:17.956648 26814 fs_manager.cc:359] Metadata directory not provided
I20260929 02:57:17.956770 26814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:57:17.957072 26814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:57:17 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:57:17.977013 26814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "33c53d398a504af59dcb0fddae55337d"
format_stamp: "Formatted at 2026-09-29 02:57:17 on dist-test-slave-q5f0"
server_key: "ffee480a2514379af2a6990fc9ef91c3"
server_key_iv: "bf7fd021180b2e4d2c26c3860e5cfedb"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:17.978137 26814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "33c53d398a504af59dcb0fddae55337d"
format_stamp: "Formatted at 2026-09-29 02:57:17 on dist-test-slave-q5f0"
server_key: "ffee480a2514379af2a6990fc9ef91c3"
server_key_iv: "bf7fd021180b2e4d2c26c3860e5cfedb"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:17.985993 26814 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 02:57:18.009680 26834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:57:18.011636 26814 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 02:57:18.011852 26814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "33c53d398a504af59dcb0fddae55337d"
format_stamp: "Formatted at 2026-09-29 02:57:17 on dist-test-slave-q5f0"
server_key: "ffee480a2514379af2a6990fc9ef91c3"
server_key_iv: "bf7fd021180b2e4d2c26c3860e5cfedb"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:18.012153 26814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:57:18.043344 26814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:57:18.048825 26814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:57:18.049309 26814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:57:18.050766 26814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:57:18.052589 26814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:57:18.052686 26814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:57:18.052806 26814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:57:18.052860 26814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:57:18.166780 26814 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:34037
I20260929 02:57:18.166800 26946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:34037 every 8 connection(s)
I20260929 02:57:18.168444 26814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:57:18.172740 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 26814
I20260929 02:57:18.172974 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:57:18.173841 15433 external_mini_cluster.cc:1428] Setting key d5c462200f3e1db0d88cb325e3c5bbe9
Sep 29 02:57:18 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
I20260929 02:57:18.223400 26950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:18.180279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:39187 (local address 127.15.18.65:40213)
0929 02:57:18.189279 (+ 9000us) negotiation.cc:107] Waiting for socket to connect
0929 02:57:18.189336 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:57:18.191860 (+ 2524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:57:18.193122 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:57:18.193137 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:57:18.193807 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:57:18.194742 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:18.194773 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:18.203749 (+ 8976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:18.203758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:18.206033 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:18.206062 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:18.207797 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:18.207804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:18.208151 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:18.209010 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:57:18.209040 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:57:18.213208 (+ 4168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:57:18.217522 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:57:18.217534 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:57:18.217556 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:57:18.218237 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:57:18.218703 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:57:18.218708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:57:18.218712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:57:18.219013 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:57:18.219754 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:57:18.219763 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:57:18.220231 (+ 468us) client_negotiation.cc:769] Sending connection context
0929 02:57:18.221416 (+ 1185us) client_negotiation.cc:240] Negotiation successful
0929 02:57:18.221989 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8623,"thread_start_us":8313,"threads_started":1}
I20260929 02:57:18.224830 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:18.179470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:40213 (local address 127.15.18.126:39187)
0929 02:57:18.187924 (+ 8454us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:18.187931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:18.189495 (+ 1564us) server_negotiation.cc:407] Connection header received
0929 02:57:18.192389 (+ 2894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:18.192395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:18.192505 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:18.192614 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:57:18.194995 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:18.196215 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:18.206593 (+ 10378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:18.207237 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:18.207353 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:18.213583 (+ 6230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:57:18.213591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:57:18.213596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:57:18.213642 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:57:18.217295 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:18.218403 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:18.218409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:18.218412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:18.218569 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:18.219151 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:18.219156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:18.219160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:18.219544 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:57:18.219669 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:18.224186 (+ 4517us) server_negotiation.cc:299] Negotiation successful
0929 02:57:18.224447 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8315,"thread_start_us":8171,"threads_started":1}
I20260929 02:57:18.226359 26947 heartbeater.cc:347] Connected to a master server at 127.15.18.126:39187
I20260929 02:57:18.226696 26947 heartbeater.cc:464] Registering TS with master...
I20260929 02:57:18.227665 26947 heartbeater.cc:511] Master 127.15.18.126:39187 requested a full tablet report, sending...
I20260929 02:57:18.230441 26734 ts_manager.cc:194] Registered new tserver with Master: 33c53d398a504af59dcb0fddae55337d (127.15.18.65:34037)
I20260929 02:57:18.232501 26734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:40213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:57:18.255985 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:39927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39187
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:57:18 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
Sep 29 02:57:18 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
W20260929 02:57:18.678174 26953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:57:18.678683 26953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:57:18.678862 26953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:57:18.690186 26953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:57:18.690415 26953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:57:18.690644 26953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:57:18.706208 26953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.18.84:39927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:57:18.708158 26953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:57:18.709975 26953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:57:18.747361 26960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:57:18.747658 26961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:57:18.752894 26963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:57:18.760299 26953 server_base.cc:1034] running on GCE node
I20260929 02:57:18.761344 26953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:57:18.775774 26953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:57:18.777094 26953 hybrid_clock.cc:648] HybridClock initialized: now 1790650638777026 us; error 75 us; skew 500 ppm
I20260929 02:57:18.777535 26953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:57:18 dist-test-slave-q5f0 krb5kdc[26258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650638, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:57:18.781807 26953 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:57:18.808295 26953 webserver.cc:460] Webserver started at http://127.15.18.66:33183/ using document root <none> and password file <none>
I20260929 02:57:18.809832 26953 fs_manager.cc:359] Metadata directory not provided
I20260929 02:57:18.810078 26953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:57:18.810542 26953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:57:18 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650638, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:57:18.855165 26953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "cd3b08fc8e0b42e7a87320dbed85db0f"
format_stamp: "Formatted at 2026-09-29 02:57:18 on dist-test-slave-q5f0"
server_key: "2853866a1f0a9c75eadf9fab8b3a6db3"
server_key_iv: "cff1f303667b93e0e160242b66e2c44d"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:18.862449 26953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "cd3b08fc8e0b42e7a87320dbed85db0f"
format_stamp: "Formatted at 2026-09-29 02:57:18 on dist-test-slave-q5f0"
server_key: "2853866a1f0a9c75eadf9fab8b3a6db3"
server_key_iv: "cff1f303667b93e0e160242b66e2c44d"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:18.885159 26953 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.008s sys 0.002s
I20260929 02:57:18.923977 26977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:57:18.925938 26953 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.001s
I20260929 02:57:18.926163 26953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "cd3b08fc8e0b42e7a87320dbed85db0f"
format_stamp: "Formatted at 2026-09-29 02:57:18 on dist-test-slave-q5f0"
server_key: "2853866a1f0a9c75eadf9fab8b3a6db3"
server_key_iv: "cff1f303667b93e0e160242b66e2c44d"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:18.926415 26953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:57:18.994902 26953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:57:19.000517 26953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:57:19.000829 26953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:57:19.008217 26953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:57:19.016506 26953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:57:19.016623 26953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:57:19.016745 26953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:57:19.016805 26953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:57:19.149127 26953 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:34319
I20260929 02:57:19.149142 27089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:34319 every 8 connection(s)
I20260929 02:57:19.150418 26953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:57:19.157359 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 26953
I20260929 02:57:19.157572 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:57:19.158135 15433 external_mini_cluster.cc:1428] Setting key 0279ac403520b65fc0f5b581a1104799
Sep 29 02:57:19 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650638, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:57:19.182204 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.154914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:37113 (local address 127.15.18.126:39187)
0929 02:57:19.159694 (+ 4780us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:19.159700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:19.159719 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:57:19.159775 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:19.159780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:19.159857 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:19.159972 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:57:19.162172 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.163255 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.165770 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.166357 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.166476 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.174767 (+ 8291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:57:19.174775 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:57:19.174779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:57:19.174820 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:57:19.178244 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:19.179321 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:19.179327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:19.179331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:19.179487 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:19.180085 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:19.180091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:19.180095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:19.180462 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:57:19.180590 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:19.181585 (+ 995us) server_negotiation.cc:299] Negotiation successful
0929 02:57:19.181822 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4632,"thread_start_us":4498,"threads_started":1}
I20260929 02:57:19.184831 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.154181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:39187 (local address 127.15.18.66:37113)
0929 02:57:19.155758 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0929 02:57:19.155820 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 02:57:19.158582 (+ 2762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:57:19.160431 (+ 1849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:57:19.160445 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:57:19.161096 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:57:19.161955 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.161983 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.163476 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.163482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.165574 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.165592 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.166780 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.166788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.167104 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.171751 (+ 4647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:57:19.171784 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:57:19.174452 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:57:19.178473 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:57:19.178484 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:57:19.178504 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:57:19.179166 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:57:19.179665 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:57:19.179670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:57:19.179673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:57:19.179955 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:57:19.180697 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:57:19.180705 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:57:19.181160 (+ 455us) client_negotiation.cc:769] Sending connection context
0929 02:57:19.182835 (+ 1675us) client_negotiation.cc:240] Negotiation successful
0929 02:57:19.183398 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"thread_start_us":985,"threads_started":1}
I20260929 02:57:19.186753 27090 heartbeater.cc:347] Connected to a master server at 127.15.18.126:39187
I20260929 02:57:19.187067 27090 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
I20260929 02:57:19.187974 27090 heartbeater.cc:511] Master 127.15.18.126:39187 requested a full tablet report, sending...
I20260929 02:57:19.189805 26734 ts_manager.cc:194] Registered new tserver with Master: cd3b08fc8e0b42e7a87320dbed85db0f (127.15.18.66:34319)
I20260929 02:57:19.190874 26734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:37113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:57:19.225447 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:39927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39187
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:57:19.236464 26947 heartbeater.cc:503] Master 127.15.18.126:39187 was elected leader, sending a full tablet report...
W20260929 02:57:19.597579 27096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:57:19.597879 27096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:57:19.597987 27096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:57:19.605276 27096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:57:19.605396 27096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:57:19.605507 27096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:57:19.615326 27096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.18.84:39927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:57:19.616626 27096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:57:19.617980 27096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:57:19.630704 27102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:57:19.630705 27101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:57:19.631174 27104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:57:19.631628 27096 server_base.cc:1034] running on GCE node
I20260929 02:57:19.632251 27096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:57:19.633178 27096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:57:19.634385 27096 hybrid_clock.cc:648] HybridClock initialized: now 1790650639634335 us; error 43 us; skew 500 ppm
I20260929 02:57:19.634693 27096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:57:19 dist-test-slave-q5f0 krb5kdc[26258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650639, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:57:19.638108 27096 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:57:19.639591 27096 webserver.cc:460] Webserver started at http://127.15.18.67:34865/ using document root <none> and password file <none>
I20260929 02:57:19.640268 27096 fs_manager.cc:359] Metadata directory not provided
I20260929 02:57:19.640391 27096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:57:19.640667 27096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:57:19 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650639, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:57:19.657820 27096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "f94673ccbf6a4deba9ba936cf9ed50ee"
format_stamp: "Formatted at 2026-09-29 02:57:19 on dist-test-slave-q5f0"
server_key: "f76d98d9cf4163f14121ec03103cf951"
server_key_iv: "8db8f4eef7cee49220d3ce8a9187f8ae"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:19.658857 27096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "f94673ccbf6a4deba9ba936cf9ed50ee"
format_stamp: "Formatted at 2026-09-29 02:57:19 on dist-test-slave-q5f0"
server_key: "f76d98d9cf4163f14121ec03103cf951"
server_key_iv: "8db8f4eef7cee49220d3ce8a9187f8ae"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:19.664377 27096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 02:57:19.679610 27111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:57:19.681268 27096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:57:19.681435 27096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "f94673ccbf6a4deba9ba936cf9ed50ee"
format_stamp: "Formatted at 2026-09-29 02:57:19 on dist-test-slave-q5f0"
server_key: "f76d98d9cf4163f14121ec03103cf951"
server_key_iv: "8db8f4eef7cee49220d3ce8a9187f8ae"
server_key_version: "kuduclusterkey@0"
I20260929 02:57:19.681666 27096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:57:19.705483 27096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:57:19.709290 27096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:57:19.709604 27096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:57:19.710584 27096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:57:19.711971 27096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:57:19.712044 27096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:57:19.712149 27096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:57:19.712195 27096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:57:19.741817 27096 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:38517
I20260929 02:57:19.741851 27223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:38517 every 8 connection(s)
I20260929 02:57:19.743178 27096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:57:19.744608 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 27096
I20260929 02:57:19.744752 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:57:19.745146 15433 external_mini_cluster.cc:1428] Setting key dd47b2f3e56b49db6b0bc6293a16d37b
Sep 29 02:57:19 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650639, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:57:19.764242 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.747102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:52757 (local address 127.15.18.126:39187)
0929 02:57:19.747678 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:19.747683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:19.748879 (+ 1196us) server_negotiation.cc:407] Connection header received
0929 02:57:19.750617 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:19.750621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:19.750685 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:19.750774 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:57:19.752202 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.752947 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.754596 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.755007 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.755109 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.758443 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:57:19.758448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:57:19.758451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:57:19.758483 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:57:19.761606 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:19.762264 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:19.762268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:19.762271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:19.762416 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:19.762777 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:19.762780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:19.762782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:19.763038 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:57:19.763146 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:19.763829 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 02:57:19.763984 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":242,"threads_started":1}
I20260929 02:57:19.765182 27227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.747883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:39187 (local address 127.15.18.67:52757)
0929 02:57:19.748707 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 02:57:19.748757 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:57:19.750356 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:57:19.750974 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:57:19.750983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:57:19.751402 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:57:19.752052 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.752070 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.753112 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.753117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.754408 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.754420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.755237 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.755242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.755491 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.756211 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:57:19.756237 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:57:19.758180 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:57:19.761721 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:57:19.761728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:57:19.761742 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:57:19.762156 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:57:19.762499 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:57:19.762502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:57:19.762505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:57:19.762680 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:57:19.763137 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:57:19.763143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:57:19.763467 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 02:57:19.763732 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 02:57:19.764118 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":320,"threads_started":1}
I20260929 02:57:19.766462 27224 heartbeater.cc:347] Connected to a master server at 127.15.18.126:39187
I20260929 02:57:19.766786 27224 heartbeater.cc:464] Registering TS with master...
I20260929 02:57:19.767452 27224 heartbeater.cc:511] Master 127.15.18.126:39187 requested a full tablet report, sending...
I20260929 02:57:19.768801 26734 ts_manager.cc:194] Registered new tserver with Master: f94673ccbf6a4deba9ba936cf9ed50ee (127.15.18.67:38517)
I20260929 02:57:19.769578 26734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:52757
I20260929 02:57:19.773403 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:57:19.789748 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.777806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.15.18.126:39187)
0929 02:57:19.777952 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:19.777958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:19.778391 (+ 433us) server_negotiation.cc:407] Connection header received
0929 02:57:19.778548 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:19.778553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:19.778623 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:19.778701 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:57:19.780065 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.780732 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.782066 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.782482 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.782586 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.784896 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:57:19.784902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:57:19.784905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:57:19.784935 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:57:19.787225 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:19.787937 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:19.787942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:19.787946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:19.788074 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:19.788424 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:19.788429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:19.788432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:19.788688 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:57:19.788914 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:19.789225 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 02:57:19.789438 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:57:19.793560 26734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:57:19.801676 26734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:57:19.832805 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.814322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:34037 (local address 127.0.0.1:58386)
0929 02:57:19.815029 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 02:57:19.815068 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:57:19.815271 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:57:19.824327 (+ 9056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:57:19.824336 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:57:19.824370 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:57:19.824854 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.824870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.826833 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.826838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.829389 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.829402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.831722 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.831728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.831929 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.831981 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 02:57:19.832106 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 02:57:19.832298 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":901,"threads_started":1}
I20260929 02:57:19.833750 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.814966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.15.18.67:38517)
0929 02:57:19.817011 (+ 2045us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:19.817022 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:19.825618 (+ 8596us) server_negotiation.cc:407] Connection header received
0929 02:57:19.825903 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:19.825912 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:19.826221 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:19.826417 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:57:19.827451 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.829039 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.830924 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.832391 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.832509 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.832722 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:19.833267 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 02:57:19.833434 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1862,"thread_start_us":1459,"threads_started":1}
I20260929 02:57:19.833801 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.814631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.15.18.65:34037)
0929 02:57:19.823734 (+ 9103us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:19.823740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:19.823773 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:57:19.823860 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:19.823865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:19.824082 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:19.824277 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:57:19.825098 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.826684 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.829581 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.831393 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.831551 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.833229 (+ 1678us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:19.833303 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 02:57:19.833468 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8932,"thread_start_us":8741,"threads_started":1}
I20260929 02:57:19.834425 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.814433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:38517 (local address 127.0.0.1:57246)
0929 02:57:19.825491 (+ 11058us) negotiation.cc:107] Waiting for socket to connect
0929 02:57:19.825521 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 02:57:19.825730 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:57:19.826764 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:57:19.826768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:57:19.826794 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:57:19.827246 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.827258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.829268 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.829273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.830759 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.830773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.832875 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.832881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.833064 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.833092 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:57:19.833740 (+ 648us) client_negotiation.cc:240] Negotiation successful
0929 02:57:19.834138 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10927,"thread_start_us":16585,"threads_started":1}
I20260929 02:57:19.837939 27159 tablet_service.cc:1467] Processing CreateTablet for tablet d73830a09efd48b2968054b96359bbde (DEFAULT_TABLE table=test-table [id=c9d43bf0da85426f8ca26f263831eb5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:57:19.837939 26882 tablet_service.cc:1467] Processing CreateTablet for tablet d73830a09efd48b2968054b96359bbde (DEFAULT_TABLE table=test-table [id=c9d43bf0da85426f8ca26f263831eb5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:57:19.838706 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.816809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.15.18.66:34319)
0929 02:57:19.824907 (+ 8098us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:19.824917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:19.831488 (+ 6571us) server_negotiation.cc:407] Connection header received
0929 02:57:19.831668 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:19.831676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:19.831976 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:19.832186 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:57:19.833064 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.834357 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.836333 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.837382 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.837496 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.837699 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:19.838240 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 02:57:19.838409 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7922,"thread_start_us":7821,"threads_started":1}
I20260929 02:57:19.839316 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.816720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:34319 (local address 127.0.0.1:50774)
0929 02:57:19.831341 (+ 14621us) negotiation.cc:107] Waiting for socket to connect
0929 02:57:19.831362 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:57:19.831497 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:57:19.832449 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:57:19.832453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:57:19.832475 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:57:19.832864 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.832878 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.834495 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.834500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.836181 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.836193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.837565 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.837571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.837797 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.837829 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:57:19.838899 (+ 1070us) client_negotiation.cc:240] Negotiation successful
0929 02:57:19.839028 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14433,"thread_start_us":8471,"threads_started":1}
I20260929 02:57:19.839582 26882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73830a09efd48b2968054b96359bbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:57:19.839561 27159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73830a09efd48b2968054b96359bbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:57:19.846781 27025 tablet_service.cc:1467] Processing CreateTablet for tablet d73830a09efd48b2968054b96359bbde (DEFAULT_TABLE table=test-table [id=c9d43bf0da85426f8ca26f263831eb5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:57:19.848039 27025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73830a09efd48b2968054b96359bbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:57:19.855662 27243 tablet_bootstrap.cc:492] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: Bootstrap starting.
I20260929 02:57:19.856365 27244 tablet_bootstrap.cc:492] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: Bootstrap starting.
I20260929 02:57:19.859107 27244 tablet_bootstrap.cc:654] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: Neither blocks nor log segments found. Creating new log.
I20260929 02:57:19.859325 27243 tablet_bootstrap.cc:654] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: Neither blocks nor log segments found. Creating new log.
I20260929 02:57:19.860113 27245 tablet_bootstrap.cc:492] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: Bootstrap starting.
I20260929 02:57:19.860373 27244 log.cc:824] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: Log is configured to *not* fsync() on all Append() calls
I20260929 02:57:19.860595 27243 log.cc:824] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:57:19.863332 27245 tablet_bootstrap.cc:654] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: Neither blocks nor log segments found. Creating new log.
I20260929 02:57:19.863566 27243 tablet_bootstrap.cc:492] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: No bootstrap required, opened a new log
I20260929 02:57:19.863564 27244 tablet_bootstrap.cc:492] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: No bootstrap required, opened a new log
I20260929 02:57:19.863834 27244 ts_tablet_manager.cc:1397] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 02:57:19.863834 27243 ts_tablet_manager.cc:1397] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 02:57:19.864334 27245 log.cc:824] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:57:19.867137 27245 tablet_bootstrap.cc:492] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: No bootstrap required, opened a new log
I20260929 02:57:19.867369 27245 ts_tablet_manager.cc:1397] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260929 02:57:19.874171 27245 raft_consensus.cc:348] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c53d398a504af59dcb0fddae55337d" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34037 } } peers { permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 } } peers { permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 } }
I20260929 02:57:19.874202 27244 raft_consensus.cc:348] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c53d398a504af59dcb0fddae55337d" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34037 } } peers { permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 } } peers { permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 } }
I20260929 02:57:19.874202 27243 raft_consensus.cc:348] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c53d398a504af59dcb0fddae55337d" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34037 } } peers { permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 } } peers { permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 } }
I20260929 02:57:19.874584 27245 raft_consensus.cc:374] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:57:19.874627 27243 raft_consensus.cc:374] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:57:19.874627 27244 raft_consensus.cc:374] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:57:19.874679 27245 raft_consensus.cc:729] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3b08fc8e0b42e7a87320dbed85db0f, State: Initialized, Role: FOLLOWER
I20260929 02:57:19.874722 27244 raft_consensus.cc:729] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94673ccbf6a4deba9ba936cf9ed50ee, State: Initialized, Role: FOLLOWER
I20260929 02:57:19.874715 27243 raft_consensus.cc:729] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c53d398a504af59dcb0fddae55337d, State: Initialized, Role: FOLLOWER
I20260929 02:57:19.875339 27245 consensus_queue.cc:260] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c53d398a504af59dcb0fddae55337d" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34037 } } peers { permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 } } peers { permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 } }
I20260929 02:57:19.875347 27244 consensus_queue.cc:260] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c53d398a504af59dcb0fddae55337d" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34037 } } peers { permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 } } peers { permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 } }
I20260929 02:57:19.875347 27243 consensus_queue.cc:260] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c53d398a504af59dcb0fddae55337d" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34037 } } peers { permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 } } peers { permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 } }
I20260929 02:57:19.877187 27090 heartbeater.cc:503] Master 127.15.18.126:39187 was elected leader, sending a full tablet report...
I20260929 02:57:19.877460 27243 ts_tablet_manager.cc:1428] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 02:57:19.877861 27245 ts_tablet_manager.cc:1428] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 02:57:19.877936 27224 heartbeater.cc:503] Master 127.15.18.126:39187 was elected leader, sending a full tablet report...
I20260929 02:57:19.879483 27244 ts_tablet_manager.cc:1428] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260929 02:57:19.909571 27091 tablet.cc:2369] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:57:19.940129 26948 tablet.cc:2369] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:57:19.950987 27249 raft_consensus.cc:482] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:57:19.951265 27249 raft_consensus.cc:504] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c53d398a504af59dcb0fddae55337d" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34037 } } peers { permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 } } peers { permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 } }
I20260929 02:57:19.952996 27249 leader_election.cc:288] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f94673ccbf6a4deba9ba936cf9ed50ee (127.15.18.67:38517), cd3b08fc8e0b42e7a87320dbed85db0f (127.15.18.66:34319)
I20260929 02:57:19.959981 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.953668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:48349 (local address 127.15.18.66:34319)
0929 02:57:19.953850 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:19.953856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:19.954680 (+ 824us) server_negotiation.cc:407] Connection header received
0929 02:57:19.954896 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:19.954902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:19.954963 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:19.955013 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:57:19.955727 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.956455 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.958335 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.959077 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.959212 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.959283 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:19.959607 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 02:57:19.959739 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 02:57:19.960101 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.953315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:34319 (local address 127.15.18.65:48349)
0929 02:57:19.954588 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0929 02:57:19.954606 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:57:19.954726 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:57:19.955174 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:57:19.955177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:57:19.955207 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:57:19.955574 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.955585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.956581 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.956590 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.958177 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.958192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.959266 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.959272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.959429 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.959452 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:57:19.959581 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 02:57:19.959708 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":52,"thread_start_us":648,"threads_started":1}
I20260929 02:57:19.960178 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.953407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:57403 (local address 127.15.18.67:38517)
0929 02:57:19.953707 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:19.953714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:19.954616 (+ 902us) server_negotiation.cc:407] Connection header received
0929 02:57:19.954819 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:19.954826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:19.954911 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:19.955000 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:57:19.955622 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.956388 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.958373 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.959161 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.959320 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.959415 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:19.959770 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 02:57:19.959899 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:57:19.960565 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:19.953273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:38517 (local address 127.15.18.65:57403)
0929 02:57:19.954479 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0929 02:57:19.954506 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:57:19.954680 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:57:19.955025 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:57:19.955030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:57:19.955056 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:57:19.955447 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.955458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.956530 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.956536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.958199 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:57:19.958212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:57:19.959387 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:57:19.959394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:57:19.959609 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:19.959634 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:57:19.959719 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 02:57:19.959900 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":40,"thread_start_us":746,"threads_started":1}
I20260929 02:57:19.961184 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73830a09efd48b2968054b96359bbde" candidate_uuid: "33c53d398a504af59dcb0fddae55337d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" is_pre_election: true
I20260929 02:57:19.961483 27179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73830a09efd48b2968054b96359bbde" candidate_uuid: "33c53d398a504af59dcb0fddae55337d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" is_pre_election: true
I20260929 02:57:19.961640 27045 raft_consensus.cc:2454] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33c53d398a504af59dcb0fddae55337d in term 0.
I20260929 02:57:19.961928 27179 raft_consensus.cc:2454] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33c53d398a504af59dcb0fddae55337d in term 0.
I20260929 02:57:19.962268 26837 leader_election.cc:302] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c53d398a504af59dcb0fddae55337d, cd3b08fc8e0b42e7a87320dbed85db0f; no voters:
I20260929 02:57:19.962637 27249 raft_consensus.cc:2781] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:57:19.962793 27249 raft_consensus.cc:482] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:57:19.962884 27249 raft_consensus.cc:3037] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:57:19.965013 27249 raft_consensus.cc:504] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c53d398a504af59dcb0fddae55337d" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34037 } } peers { permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 } } peers { permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 } }
I20260929 02:57:19.965529 27249 leader_election.cc:288] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [CANDIDATE]: Term 1 election: Requested vote from peers f94673ccbf6a4deba9ba936cf9ed50ee (127.15.18.67:38517), cd3b08fc8e0b42e7a87320dbed85db0f (127.15.18.66:34319)
I20260929 02:57:19.965974 27045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73830a09efd48b2968054b96359bbde" candidate_uuid: "33c53d398a504af59dcb0fddae55337d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f"
I20260929 02:57:19.966012 27179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73830a09efd48b2968054b96359bbde" candidate_uuid: "33c53d398a504af59dcb0fddae55337d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee"
I20260929 02:57:19.966153 27045 raft_consensus.cc:3037] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:57:19.966174 27179 raft_consensus.cc:3037] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:57:19.968287 27045 raft_consensus.cc:2454] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33c53d398a504af59dcb0fddae55337d in term 1.
I20260929 02:57:19.968504 27179 raft_consensus.cc:2454] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33c53d398a504af59dcb0fddae55337d in term 1.
I20260929 02:57:19.968695 26837 leader_election.cc:302] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c53d398a504af59dcb0fddae55337d, cd3b08fc8e0b42e7a87320dbed85db0f; no voters:
I20260929 02:57:19.969018 27249 raft_consensus.cc:2781] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:57:19.969475 27249 raft_consensus.cc:686] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 1 LEADER]: Becoming Leader. State: Replica: 33c53d398a504af59dcb0fddae55337d, State: Running, Role: LEADER
I20260929 02:57:19.970033 27249 consensus_queue.cc:237] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c53d398a504af59dcb0fddae55337d" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34037 } } peers { permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 } } peers { permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 } }
I20260929 02:57:19.974641 26733 catalog_manager.cc:5495] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d reported cstate change: term changed from 0 to 1, leader changed from <none> to 33c53d398a504af59dcb0fddae55337d (127.15.18.65). New cstate: current_term: 1 leader_uuid: "33c53d398a504af59dcb0fddae55337d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c53d398a504af59dcb0fddae55337d" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 } health_report { overall_health: UNKNOWN } } }
W20260929 02:57:19.994709 27225 tablet.cc:2369] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:57:20.020856 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:20.014833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.15.18.65:34037)
0929 02:57:20.015033 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:20.015038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:20.015080 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:57:20.015408 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:20.015414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:20.015561 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:20.015724 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:57:20.016178 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:20.016962 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:20.018525 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:20.018924 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:20.019021 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:20.019725 (+ 704us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:57:20.020253 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:57:20.020365 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:20.020520 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 02:57:20.020594 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:57:20.029206 27045 raft_consensus.cc:1261] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f [term 1 FOLLOWER]: Refusing update from remote peer 33c53d398a504af59dcb0fddae55337d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:57:20.029229 27179 raft_consensus.cc:1261] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee [term 1 FOLLOWER]: Refusing update from remote peer 33c53d398a504af59dcb0fddae55337d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:57:20.030128 27254 consensus_queue.cc:1035] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3b08fc8e0b42e7a87320dbed85db0f" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:57:20.030463 27249 consensus_queue.cc:1035] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f94673ccbf6a4deba9ba936cf9ed50ee" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:57:20.042997 27258 mvcc.cc:205] Tried to move back new op lower bound from 7334505021549600768 to 7334505021320802304. Current Snapshot: MvccSnapshot[applied={T|T < 7334505021549600768}]
I20260929 02:57:20.052541 26731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:57:20.052870 26731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:57:20.056957 26731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a5f27948dd24b57b71d2da2e19ca36d: Sending DeleteTablet for 3 replicas of tablet d73830a09efd48b2968054b96359bbde
I20260929 02:57:20.058205 27159 tablet_service.cc:1514] Processing DeleteTablet for tablet d73830a09efd48b2968054b96359bbde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:57:20 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:57246
I20260929 02:57:20.058214 26882 tablet_service.cc:1514] Processing DeleteTablet for tablet d73830a09efd48b2968054b96359bbde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:57:20 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:58386
I20260929 02:57:20.059257 27266 tablet_replica.cc:330] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: stopping tablet replica
I20260929 02:57:20.059190 27025 tablet_service.cc:1514] Processing DeleteTablet for tablet d73830a09efd48b2968054b96359bbde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:57:20 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:50774
I20260929 02:57:20.060009 27267 tablet_replica.cc:330] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: stopping tablet replica
I20260929 02:57:20.060608 27267 raft_consensus.cc:2229] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:57:20.060624 27266 raft_consensus.cc:2229] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:57:20.061290 27267 raft_consensus.cc:2258] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:57:20.061353 27266 raft_consensus.cc:2258] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:57:20.061800 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 26814
I20260929 02:57:20.063932 27267 ts_tablet_manager.cc:1905] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:57:20.065871 27266 ts_tablet_manager.cc:1905] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:57:20.073957 27266 ts_tablet_manager.cc:1918] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:57:20.076233 27268 tablet_replica.cc:330] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: stopping tablet replica
I20260929 02:57:20.076725 27268 raft_consensus.cc:2229] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:57:20.077267 27268 raft_consensus.cc:2258] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:57:20.079561 27268 ts_tablet_manager.cc:1905] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:57:20.081607 27267 ts_tablet_manager.cc:1918] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:57:20.081861 27267 log.cc:1196] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/d73830a09efd48b2968054b96359bbde
I20260929 02:57:20.082468 27267 ts_tablet_manager.cc:1939] T d73830a09efd48b2968054b96359bbde P f94673ccbf6a4deba9ba936cf9ed50ee: Deleting consensus metadata
I20260929 02:57:20.084705 26721 catalog_manager.cc:4841] TS f94673ccbf6a4deba9ba936cf9ed50ee (127.15.18.67:38517): tablet d73830a09efd48b2968054b96359bbde (table test-table [id=c9d43bf0da85426f8ca26f263831eb5a]) successfully deleted
Sep 29 02:57:20 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:57:20.085232 27266 log.cc:1196] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/d73830a09efd48b2968054b96359bbde
I20260929 02:57:20.085985 27266 ts_tablet_manager.cc:1939] T d73830a09efd48b2968054b96359bbde P 33c53d398a504af59dcb0fddae55337d: Deleting consensus metadata
I20260929 02:57:20.086277 27268 ts_tablet_manager.cc:1918] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:57:20.086422 27268 log.cc:1196] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/d73830a09efd48b2968054b96359bbde
I20260929 02:57:20.086882 27268 ts_tablet_manager.cc:1939] T d73830a09efd48b2968054b96359bbde P cd3b08fc8e0b42e7a87320dbed85db0f: Deleting consensus metadata
I20260929 02:57:20.088140 26718 catalog_manager.cc:4841] TS 33c53d398a504af59dcb0fddae55337d (127.15.18.65:34037): tablet d73830a09efd48b2968054b96359bbde (table test-table [id=c9d43bf0da85426f8ca26f263831eb5a]) successfully deleted
I20260929 02:57:20.088577 26720 catalog_manager.cc:4841] TS cd3b08fc8e0b42e7a87320dbed85db0f (127.15.18.66:34319): tablet d73830a09efd48b2968054b96359bbde (table test-table [id=c9d43bf0da85426f8ca26f263831eb5a]) successfully deleted
I20260929 02:57:20.093194 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:20.063016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.15.18.65:34037)
0929 02:57:20.063221 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:20.063227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:20.075260 (+ 12033us) server_negotiation.cc:407] Connection header received
0929 02:57:20.075477 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:20.075482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:20.075606 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:20.075697 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:57:20.077303 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:20.078423 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:20.080305 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:20.080944 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:20.081043 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:20.086805 (+ 5762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:57:20.086820 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:57:20.086840 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:57:20.086900 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:57:20.090408 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:20.091106 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:20.091115 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:20.091118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:20.091267 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:20.091674 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:20.091678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:20.091680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:20.092039 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:57:20.092323 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:20.092641 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 02:57:20.092870 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:57:20.093863 26942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:57:20.203730 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 26814
I20260929 02:57:20.219290 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 26953
Sep 29 02:57:20 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:57:20.235216 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:20.220192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.15.18.66:34319)
0929 02:57:20.220351 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:20.220354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:20.220372 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:57:20.220482 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:20.220487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:20.220559 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:20.220642 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:57:20.221878 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:20.222637 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:20.224182 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:20.224647 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:20.224756 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:20.229442 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:57:20.229462 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:57:20.229478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:57:20.229517 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:57:20.232435 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:20.233211 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:20.233219 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:20.233221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:20.233357 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:20.233766 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:20.233769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:20.233771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:20.234150 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:57:20.234366 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:20.234761 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 02:57:20.234948 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:57:20.235939 27085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:57:20.324410 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 26953
I20260929 02:57:20.337265 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 27096
Sep 29 02:57:20 dist-test-slave-q5f0 krb5kdc[26258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:57:20.352687 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:57:20.338047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.15.18.67:38517)
0929 02:57:20.338251 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:57:20.338255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:57:20.338277 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:57:20.338448 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:57:20.338452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:57:20.338515 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:57:20.338594 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:57:20.339940 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:20.340662 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:20.342060 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:57:20.342452 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:57:20.342552 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:57:20.347084 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:57:20.347099 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:57:20.347116 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:57:20.347152 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:57:20.349895 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:20.350651 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:20.350659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:20.350661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:20.350797 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:57:20.351187 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:57:20.351190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:57:20.351192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:57:20.351605 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:57:20.351812 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 02:57:20.352175 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 02:57:20.352342 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:57:20.353365 27219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:57:20.441726 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 27096
I20260929 02:57:20.454753 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 26702
I20260929 02:57:20.455931 26764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:57:20.566555 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 26702
I20260929 02:57:20.588260 15433 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 02:57:20.923975 15433 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 02:57:20.924201 15433 mini_ranger.cc:66] Stopping Ranger...
I20260929 02:57:21.260038 15433 mini_ranger.cc:68] Stopped Ranger
2026-09-29 02:57:21.260 UTC [26288] LOG: received smart shutdown request
2026-09-29 02:57:21.263 UTC [26288] LOG: background worker "logical replication launcher" (PID 26297) exited with exit code 1
2026-09-29 02:57:21.264 UTC [26292] LOG: shutting down
2026-09-29 02:57:21.439 UTC [26288] LOG: database system is shut down
2026-09-29T02:57:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49323 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:57:21 dist-test-slave-q5f0 krb5kdc[27282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:57:21 dist-test-slave-q5f0 krb5kdc[27282](info): set up 2 sockets
Sep 29 02:57:21 dist-test-slave-q5f0 krb5kdc[27282](info): commencing operation
krb5kdc: starting...
W20260929 02:57:22.556020 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:57:22 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:57:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/rangeradmin_127.15.18.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/rangeradmin_127.15.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/rangerlookup_127.15.18.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/rangerlookup_127.15.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/HTTP_127.15.18.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/HTTP_127.15.18.84@KRBTEST.COM.keytab.
I20260929 02:57:22.867290 15433 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxb3wIY/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/postgres -l logfile start
2026-09-29 02:57:25.401 UTC [27312] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 02:57:25.401 UTC [27312] LOG: listening on IPv4 address "127.15.18.84", port 47193
2026-09-29 02:57:25.405 UTC [27312] LOG: listening on Unix socket "/tmp/.s.PGSQL.47193"
2026-09-29 02:57:25.425 UTC [27315] LOG: database system was shut down at 2026-09-29 02:57:23 UTC
2026-09-29 02:57:25.429 UTC [27312] LOG: database system is ready to accept connections
I20260929 02:57:26.319708 15433 mini_postgres.cc:96] Postgres bound to 47193
2026-09-29 02:57:26.339 UTC [27323] FATAL: database "slave" does not exist
127.15.18.84:47193 - accepting connections
I20260929 02:57:26.339771 15433 mini_ranger.cc:161] Starting Ranger...
I20260929 02:57:26.364181 15433 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 02:57:26.871031 15433 mini_ranger.cc:87] Created ranger Postgres database
I20260929 02:57:26.871124 15433 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-admin
2026-09-29 02:57:27,149 [I] DB FLAVOR :POSTGRES
2026-09-29 02:57:27,149 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:57:27,149 [I] Checking connection..
2026-09-29 02:57:27,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 02:57:27,400 [I] Checking connection passed.
2026-09-29 02:57:27,400 [I] --------- Verifying version history table ---------
2026-09-29 02:57:27,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:57:27,664 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:57:27,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:57:27,916 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:57:27,916 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 02:57:27,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 02:57:28.177 UTC [27387] WARNING: there is no transaction in progress
2026-09-29 02:57:28,183 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 02:57:28,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:57:28,426 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 02:57:28,426 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:57:28,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 02:57:28,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 02:57:28,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q5f0.c.gcp-upstream.internal','N') ;"
2026-09-29 02:57:29,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:57:29,454 [I] Table x_portal_user does not exist in database ranger
2026-09-29 02:57:29,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:57:29,698 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 02:57:29,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:57:29,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:57:30,170 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 02:57:30,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 02:57:31.175 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.193 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.210 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.227 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.244 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.262 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.325 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.332 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.348 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.359 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.370 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.385 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.394 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.401 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.408 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:31.414 UTC [27495] WARNING: there is no transaction in progress
2026-09-29 02:57:32,076 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 02:57:32,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 02:57:32,299 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 02:57:32,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:57:32,531 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 02:57:32,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:57:32,791 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 02:57:32,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:57:33,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:57:33,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q5f0.c.gcp-upstream.internal';"
2026-09-29 02:57:33,513 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 02:57:33,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 02:57:33,776 [I] DB_PATCHES have already been applied
I20260929 02:57:33.782114 15433 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxb3wIY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 02:57:33.782207 15433 mini_ranger.cc:193] Using host: 127.15.18.84
I20260929 02:57:33.795697 15433 mini_ranger.cc:227] Ranger admin URL: http://127.15.18.84:44525
Sep 29, 2026 2:57:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:57:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:57:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:57:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:57:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:57:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:57:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 02:57:34.817665 15433 mini_ranger.cc:160] Time spent starting Ranger: real 8.478s user 0.004s sys 0.033s
Sep 29, 2026 2:57:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:57:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 2:57:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:57:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:57:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 02:57:56.985733 15433 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.18.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.18.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/rangerkms_127.15.18.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.18.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/rangerkms_127.15.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.18.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.18.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.18.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/HTTP_127.15.18.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.18.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/HTTP_127.15.18.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 02:57:57 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650677, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 02:57:57.156672 15433 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 02:57:57.176121 15433 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 02:57:58.053973 15433 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 02:57:58.054066 15433 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-kms
I20260929 02:57:58.054105 15433 mini_ranger_kms.cc:227] Using postgres at 127.15.18.84:47193
2026-09-29 02:57:58,392 [I] DB FLAVOR :POSTGRES
2026-09-29 02:57:58,392 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:57:58,392 [I] Checking connection
2026-09-29 02:57:58,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 02:57:58,617 [I] connection success
2026-09-29 02:57:58,617 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 02:57:58,617 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 02:57:58,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 02:57:58,853 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 02:57:58,853 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:57:58,853 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 02:57:58,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.18.84:47193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 02:57:59,120 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 02:57:59.579062 15433 mini_ranger_kms.cc:310] Created kms service
I20260929 02:57:59.794893 15433 mini_ranger_kms.cc:326] Created kudu user
I20260929 02:57:59.852255 15433 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 02:57:59 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650677, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:00.255767 15433 mini_ranger_kms.cc:384] Added ranger policy
I20260929 02:58:00.256129 15433 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxb3wIY/build/asan/bin/postgresql.jar:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxb3wIY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 02:58:00.256206 15433 mini_ranger_kms.cc:240] Using host: 127.15.18.84
I20260929 02:58:00.272627 15433 mini_ranger_kms.cc:276] Ranger KMS PID: 27675
I20260929 02:58:00.272755 15433 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.18.84:46663
Sep 29, 2026 2:58:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:58:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:58:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:58:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:58:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:58:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 02:58:02.364130 15433 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.207s user 0.093s sys 0.265s
I20260929 02:58:02.364372 15433 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 02:58:02.364535 15433 mini_ranger_kms.cc:401] 127.15.18.84:46663/kms/v1/keys
Sep 29 02:58:02 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650682, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.18.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:58:02 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650682, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.18.84@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
Sep 29 02:58:03 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650682, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.18.84@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
Sep 29 02:58:05 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:58:05.491549 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:38719
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:38719
--ranger_config_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:46663/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:05.813367 27726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:05.813669 27726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:05.813782 27726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:05.820856 27726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:58:05.820999 27726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:05.821074 27726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:05.821151 27726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:58:05.821214 27726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:58:05.831115 27726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.18.84:46663/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:38719
--ranger_config_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:38719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:05.832382 27726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:05.833666 27726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:05.845005 27731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:05.845069 27732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:05.844918 27734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:05.845685 27726 server_base.cc:1034] running on GCE node
I20260929 02:58:05.846379 27726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:05.847767 27726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:05.848959 27726 hybrid_clock.cc:648] HybridClock initialized: now 1790650685848920 us; error 38 us; skew 500 ppm
I20260929 02:58:05.849279 27726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:05 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650685, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:05.855314 27726 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:58:05.856766 27726 webserver.cc:460] Webserver started at http://127.15.18.126:46809/ using document root <none> and password file <none>
I20260929 02:58:05.857388 27726 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:05.857491 27726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:05.857771 27726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:58:05 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650685, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:05.918771 27726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "a1fa5e9b79e84214bd08f2275b5f277a"
format_stamp: "Formatted at 2026-09-29 02:58:05 on dist-test-slave-q5f0"
server_key: "001f26ae512821c25d419d30fadf46cd"
server_key_iv: "07874faa984d910fed3696edfbc5dbb7"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:05.919661 27726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "a1fa5e9b79e84214bd08f2275b5f277a"
format_stamp: "Formatted at 2026-09-29 02:58:05 on dist-test-slave-q5f0"
server_key: "001f26ae512821c25d419d30fadf46cd"
server_key_iv: "07874faa984d910fed3696edfbc5dbb7"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:05.924976 27726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:58:05.944301 27741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:05.945766 27726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:58:05.945945 27726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "a1fa5e9b79e84214bd08f2275b5f277a"
format_stamp: "Formatted at 2026-09-29 02:58:05 on dist-test-slave-q5f0"
server_key: "001f26ae512821c25d419d30fadf46cd"
server_key_iv: "07874faa984d910fed3696edfbc5dbb7"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:05.946185 27726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:05.966871 27726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:05.970876 27726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:05.971145 27726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:05.987078 27726 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:38719
I20260929 02:58:05.987082 27792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:38719 every 8 connection(s)
I20260929 02:58:05.988543 27726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:58:05.992378 27793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:05.996878 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 27726
I20260929 02:58:05.997072 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:58:05.997773 15433 external_mini_cluster.cc:1428] Setting key 2a350c847b020be8776bb71ad0f56ce7
I20260929 02:58:06.003710 27793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: Bootstrap starting.
I20260929 02:58:06.007256 27793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:06.008524 27793 log.cc:824] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:06.011940 27793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: No bootstrap required, opened a new log
Sep 29 02:58:06 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:06.019722 27793 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } }
I20260929 02:58:06.020154 27793 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:06.020254 27793 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fa5e9b79e84214bd08f2275b5f277a, State: Initialized, Role: FOLLOWER
I20260929 02:58:06.020848 27793 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } }
I20260929 02:58:06.021059 27793 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:58:06.021194 27793 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:58:06.021246 27796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:06.000495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.15.18.126:38719)
0929 02:58:06.001350 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:06.001366 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:06.001416 (+ 50us) server_negotiation.cc:407] Connection header received
0929 02:58:06.003207 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:06.003230 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:06.003694 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:06.004065 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:06.005686 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:06.007003 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:06.008781 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:06.009796 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:06.009949 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:06.014106 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:06.014114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:06.014129 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:06.014172 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:06.017212 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:06.017873 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:06.017883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:06.017890 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:06.018067 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:06.018443 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:06.018446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:06.018448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:06.019002 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:06.019273 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:06.019757 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 02:58:06.020169 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":396,"threads_started":1}
I20260929 02:58:06.021687 27793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:06.024312 27793 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } }
I20260929 02:58:06.024724 27793 leader_election.cc:302] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1fa5e9b79e84214bd08f2275b5f277a; no voters:
I20260929 02:58:06.025185 27793 leader_election.cc:288] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:58:06.025306 27798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:58:06.025911 27798 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 1 LEADER]: Becoming Leader. State: Replica: a1fa5e9b79e84214bd08f2275b5f277a, State: Running, Role: LEADER
I20260929 02:58:06.026388 27798 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } }
I20260929 02:58:06.027036 27793 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:58:06.028771 27799 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } } }
I20260929 02:58:06.029126 27799 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:06.029664 27800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fa5e9b79e84214bd08f2275b5f277a. Latest consensus state: current_term: 1 leader_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } } }
I20260929 02:58:06.029942 27800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:06.036345 27803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:58:06.042186 27803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:58:06.042759 27793 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:58:06.066361 27803 catalog_manager.cc:1347] Generated new cluster ID: 265d0703c03c4ffeb104d1188baa62d6
I20260929 02:58:06.066558 27803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:58:06.077339 27803 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:58:06.079237 27803 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:58:06.093811 27803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: Generated new TSK 0
I20260929 02:58:06.095084 27803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:58:06.764916 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:46663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38719
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:58:07 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 02:58:07.132002 27838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:07.132289 27838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:07.132450 27838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:07.140180 27838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:07.140303 27838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:07.140475 27838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:58:07.150424 27838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.18.84:46663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:07.151823 27838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:07.153079 27838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:07.168783 27847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:07.169145 27848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:07.170605 27851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:07.172344 27838 server_base.cc:1034] running on GCE node
I20260929 02:58:07.173048 27838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:07.175837 27838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:07.177117 27838 hybrid_clock.cc:648] HybridClock initialized: now 1790650687177053 us; error 69 us; skew 500 ppm
I20260929 02:58:07.177507 27838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:07 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:07.180707 27838 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:58:07.184533 27838 webserver.cc:460] Webserver started at http://127.15.18.65:33397/ using document root <none> and password file <none>
I20260929 02:58:07.185276 27838 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:07.185395 27838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:07.185707 27838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:58:07 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:07.209648 27838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "83945042ea374edc8ee66384ead8ead3"
format_stamp: "Formatted at 2026-09-29 02:58:07 on dist-test-slave-q5f0"
server_key: "f7ae9f7cc43d53e64ebf903aea77e911"
server_key_iv: "6ad880f54dfa9c103a43e6bb3cb37c00"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:07.210863 27838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "83945042ea374edc8ee66384ead8ead3"
format_stamp: "Formatted at 2026-09-29 02:58:07 on dist-test-slave-q5f0"
server_key: "f7ae9f7cc43d53e64ebf903aea77e911"
server_key_iv: "6ad880f54dfa9c103a43e6bb3cb37c00"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:07.222285 27838 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.007s sys 0.003s
I20260929 02:58:07.240165 27858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:07.242076 27838 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 02:58:07.242339 27838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "83945042ea374edc8ee66384ead8ead3"
format_stamp: "Formatted at 2026-09-29 02:58:07 on dist-test-slave-q5f0"
server_key: "f7ae9f7cc43d53e64ebf903aea77e911"
server_key_iv: "6ad880f54dfa9c103a43e6bb3cb37c00"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:07.242605 27838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:07.279134 27838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:07.284324 27838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:07.284719 27838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:07.285984 27838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:07.287839 27838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:07.287964 27838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:07.288112 27838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:07.288194 27838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:07.368662 27838 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:46037
I20260929 02:58:07.368677 27970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:46037 every 8 connection(s)
I20260929 02:58:07.369998 27838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:58:07.374238 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 27838
I20260929 02:58:07.374454 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:58:07.375310 15433 external_mini_cluster.cc:1428] Setting key dd84b556ee1779cc6495ba10c05dc33b
Sep 29 02:58:07 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:07.394846 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:07.372748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:58499 (local address 127.15.18.126:38719)
0929 02:58:07.373265 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:07.373271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:07.376047 (+ 2776us) server_negotiation.cc:407] Connection header received
0929 02:58:07.376120 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:07.376125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:07.377269 (+ 1144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:07.377360 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:07.379282 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:07.380377 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:07.382798 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:07.383379 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:07.383469 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:07.387991 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:07.387997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:07.388002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:07.388043 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:07.391284 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:07.392215 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:07.392221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:07.392224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:07.392365 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:07.392894 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:07.392899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:07.392902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:07.393251 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:07.393379 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:07.394367 (+ 988us) server_negotiation.cc:299] Negotiation successful
0929 02:58:07.394580 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":240,"threads_started":1}
I20260929 02:58:07.397094 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:07.373240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38719 (local address 127.15.18.65:58499)
0929 02:58:07.373974 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:07.374038 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:07.375739 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:07.377756 (+ 2017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:07.377768 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:07.378354 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:07.379099 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:07.379121 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:07.380576 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:07.380581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:07.382628 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:07.382642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:07.383848 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:07.383856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:07.384163 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:07.385030 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:07.385058 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:07.387714 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:07.391466 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:07.391475 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:07.391494 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:07.392083 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:07.392500 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:07.392504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:07.392507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:07.392774 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:07.393473 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:07.393481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:07.393924 (+ 443us) client_negotiation.cc:769] Sending connection context
0929 02:58:07.395131 (+ 1207us) client_negotiation.cc:240] Negotiation successful
0929 02:58:07.395597 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":294,"threads_started":1}
I20260929 02:58:07.400867 27971 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38719
I20260929 02:58:07.401221 27971 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:07.402150 27971 heartbeater.cc:511] Master 127.15.18.126:38719 requested a full tablet report, sending...
I20260929 02:58:07.405036 27758 ts_manager.cc:194] Registered new tserver with Master: 83945042ea374edc8ee66384ead8ead3 (127.15.18.65:46037)
I20260929 02:58:07.407191 27758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:58499
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:58:07.451157 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:46663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38719
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:58:07 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
W20260929 02:58:07.826226 27977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:07.826705 27977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:07.826905 27977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:07.837409 27977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:07.837630 27977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:07.837857 27977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
Sep 29 02:58:07 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:07.852167 27977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.18.84:46663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:07.854034 27977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:07.855957 27977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:07.890566 27983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:07.892941 27985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:07.897413 27987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:07.897416 27977 server_base.cc:1034] running on GCE node
I20260929 02:58:07.899624 27977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:07.907569 27977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:07.909091 27977 hybrid_clock.cc:648] HybridClock initialized: now 1790650687908941 us; error 129 us; skew 500 ppm
I20260929 02:58:07.909631 27977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:07 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:07.914191 27977 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:58:07.944010 27977 webserver.cc:460] Webserver started at http://127.15.18.66:42139/ using document root <none> and password file <none>
I20260929 02:58:07.945151 27977 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:07.945350 27977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:07.945740 27977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:58:07 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:07.981918 27977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "97e68dcd328c4ef79520c3181cfce1b5"
format_stamp: "Formatted at 2026-09-29 02:58:07 on dist-test-slave-q5f0"
server_key: "eb10dc223e1ede566a0720730848cdca"
server_key_iv: "854b9027caca1cfbb80995553837417d"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:07.983126 27977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "97e68dcd328c4ef79520c3181cfce1b5"
format_stamp: "Formatted at 2026-09-29 02:58:07 on dist-test-slave-q5f0"
server_key: "eb10dc223e1ede566a0720730848cdca"
server_key_iv: "854b9027caca1cfbb80995553837417d"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:07.996591 27977 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.003s sys 0.008s
I20260929 02:58:08.043574 27994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:08.052798 27977 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.007s sys 0.001s
I20260929 02:58:08.053026 27977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "97e68dcd328c4ef79520c3181cfce1b5"
format_stamp: "Formatted at 2026-09-29 02:58:07 on dist-test-slave-q5f0"
server_key: "eb10dc223e1ede566a0720730848cdca"
server_key_iv: "854b9027caca1cfbb80995553837417d"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:08.053252 27977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:08.150615 27977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:08.155961 27977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:08.156250 27977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:08.157259 27977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:08.158828 27977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:08.158905 27977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:08.159055 27977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:08.159116 27977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:08.218978 27977 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:42981
I20260929 02:58:08.219062 28113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:42981 every 8 connection(s)
I20260929 02:58:08.220304 27977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:58:08.221827 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 27977
I20260929 02:58:08.221994 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:58:08.222414 15433 external_mini_cluster.cc:1428] Setting key c13af6081434f47c402d0a592262e7e0
Sep 29 02:58:08 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
I20260929 02:58:08.248670 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:08.223790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:41401 (local address 127.15.18.126:38719)
0929 02:58:08.224307 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:08.224313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:08.228961 (+ 4648us) server_negotiation.cc:407] Connection header received
0929 02:58:08.230936 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:08.230942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:08.231049 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:08.231187 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:08.232796 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.233645 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.236106 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.236679 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.236803 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:08.241198 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:08.241206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:08.241212 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:08.241257 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:08.244824 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:08.245789 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:08.245794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:08.245796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:08.245917 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:08.246479 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:08.246482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:08.246485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:08.246774 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:08.246882 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:08.248005 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0929 02:58:08.248243 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":269,"threads_started":1}
I20260929 02:58:08.249882 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:08.223950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38719 (local address 127.15.18.66:41401)
0929 02:58:08.228418 (+ 4468us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:08.228477 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:08.230607 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:08.231458 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:08.231472 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:08.231978 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:08.232614 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:08.232632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.233822 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.233828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:08.235934 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:08.235951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.236961 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.236969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:08.237302 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:08.238084 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:08.238113 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:08.240862 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:08.245012 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:08.245023 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:08.245042 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:08.245651 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:08.246031 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:08.246068 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:08.246072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:08.246359 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:08.246903 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:08.246948 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:08.247547 (+ 599us) client_negotiation.cc:769] Sending connection context
0929 02:58:08.247871 (+ 324us) client_negotiation.cc:240] Negotiation successful
0929 02:58:08.248453 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4131,"thread_start_us":3891,"threads_started":1}
I20260929 02:58:08.251611 28114 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38719
I20260929 02:58:08.251951 28114 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:08.252781 28114 heartbeater.cc:511] Master 127.15.18.126:38719 requested a full tablet report, sending...
I20260929 02:58:08.254472 27758 ts_manager.cc:194] Registered new tserver with Master: 97e68dcd328c4ef79520c3181cfce1b5 (127.15.18.66:42981)
I20260929 02:58:08.255693 27758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:41401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:58:08.275178 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:46663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38719
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:58:08.411350 27971 heartbeater.cc:503] Master 127.15.18.126:38719 was elected leader, sending a full tablet report...
W20260929 02:58:08.609866 28120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:08.610152 28120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:08.610292 28120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:08.617465 28120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:08.617590 28120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:08.617731 28120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:58:08.627399 28120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.18.84:46663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:08.628726 28120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:08.629997 28120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:08.642465 28125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:08.642519 28126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:08.643028 28128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:08.643091 28120 server_base.cc:1034] running on GCE node
I20260929 02:58:08.643716 28120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:08.644598 28120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:08.645790 28120 hybrid_clock.cc:648] HybridClock initialized: now 1790650688645743 us; error 39 us; skew 500 ppm
I20260929 02:58:08.646078 28120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:08 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650688, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:08.649186 28120 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:58:08.650695 28120 webserver.cc:460] Webserver started at http://127.15.18.67:35995/ using document root <none> and password file <none>
I20260929 02:58:08.651361 28120 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:08.651490 28120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:08.651806 28120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:58:08 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650688, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:08.669064 28120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "3d63624ade0743be8d6408317bfa3947"
format_stamp: "Formatted at 2026-09-29 02:58:08 on dist-test-slave-q5f0"
server_key: "9a92293b0798d01b558dee2643eb8339"
server_key_iv: "2383be2102f62fd515f217e63209ea3a"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:08.670156 28120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "3d63624ade0743be8d6408317bfa3947"
format_stamp: "Formatted at 2026-09-29 02:58:08 on dist-test-slave-q5f0"
server_key: "9a92293b0798d01b558dee2643eb8339"
server_key_iv: "2383be2102f62fd515f217e63209ea3a"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:08.676968 28120 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 02:58:08.692281 28135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:08.693789 28120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:58:08.693934 28120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "3d63624ade0743be8d6408317bfa3947"
format_stamp: "Formatted at 2026-09-29 02:58:08 on dist-test-slave-q5f0"
server_key: "9a92293b0798d01b558dee2643eb8339"
server_key_iv: "2383be2102f62fd515f217e63209ea3a"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:08.694140 28120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:08.717412 28120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:08.721202 28120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:08.721475 28120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:08.722399 28120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:08.723750 28120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:08.723819 28120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:08.723912 28120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:08.723959 28120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:08.753633 28120 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:33189
I20260929 02:58:08.753664 28247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:33189 every 8 connection(s)
I20260929 02:58:08.754848 28120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:58:08.759635 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 28120
I20260929 02:58:08.759847 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:58:08.760358 15433 external_mini_cluster.cc:1428] Setting key b0b803112db2fa317fa7c40c69c1a913
Sep 29 02:58:08 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650688, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:08.774120 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:08.757386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:56725 (local address 127.15.18.126:38719)
0929 02:58:08.757816 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:08.757821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:08.758863 (+ 1042us) server_negotiation.cc:407] Connection header received
0929 02:58:08.760817 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:08.760823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:08.760906 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:08.761014 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:08.762476 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.763317 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.765022 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.765467 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.765579 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:08.768659 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:08.768664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:08.768667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:08.768700 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:08.771006 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:08.771734 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:08.771738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:08.771741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:08.771886 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:08.772364 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:08.772368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:08.772370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:08.772671 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:08.772802 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:08.773506 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 02:58:08.773779 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":211,"threads_started":1}
I20260929 02:58:08.774979 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:08.757943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38719 (local address 127.15.18.67:56725)
0929 02:58:08.758707 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:08.758759 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:08.760545 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:08.761245 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:08.761253 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:08.761741 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:08.762315 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:08.762333 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.763462 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.763466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:08.764878 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:08.764890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.765670 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.765675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:08.765870 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:08.766468 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:08.766493 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:08.768443 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:08.771135 (+ 2692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:08.771142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:08.771155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:08.771591 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:08.772057 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:08.772060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:08.772063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:08.772255 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:08.772803 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:08.772808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:08.773108 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 02:58:08.773380 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 02:58:08.773757 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":296,"threads_started":1}
I20260929 02:58:08.776343 28248 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38719
I20260929 02:58:08.776614 28248 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:08.777244 28248 heartbeater.cc:511] Master 127.15.18.126:38719 requested a full tablet report, sending...
I20260929 02:58:08.778520 27757 ts_manager.cc:194] Registered new tserver with Master: 3d63624ade0743be8d6408317bfa3947 (127.15.18.67:33189)
I20260929 02:58:08.779386 27757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:56725
I20260929 02:58:08.788466 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:58:08.790202 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 27838
Sep 29 02:58:08 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:58:08.805784 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:08.791133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.15.18.65:46037)
0929 02:58:08.791849 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:08.791856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:08.791883 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:58:08.791948 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:08.791956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:08.792160 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:08.792299 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:08.793759 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.794744 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.796161 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.796685 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.796776 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:08.800774 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:08.800785 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:08.800798 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:08.800831 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:08.803424 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:08.804081 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:08.804088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:08.804090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:08.804225 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:08.804490 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:08.804493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:08.804495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:08.804804 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:08.805010 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:08.805248 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 02:58:08.805429 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":414,"threads_started":1}
I20260929 02:58:08.806979 27966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:08.889513 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 27838
I20260929 02:58:08.901801 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 27977
Sep 29 02:58:08 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:58:08.917644 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:08.902684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.15.18.66:42981)
0929 02:58:08.903264 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:08.903272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:08.903299 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:58:08.903381 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:08.903388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:08.903668 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:08.903805 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:08.905030 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.906078 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.907372 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:08.908001 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:08.908110 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:08.912198 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:08.912209 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:08.912224 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:08.912256 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:08.914818 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:08.915489 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:08.915496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:08.915498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:08.915864 (+ 366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:08.916282 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:08.916285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:08.916287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:08.916600 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:08.916819 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:08.917140 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 02:58:08.917344 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":329,"threads_started":1}
I20260929 02:58:08.918869 28109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:09.000788 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 27977
I20260929 02:58:09.013231 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 28120
Sep 29 02:58:09 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:58:09.030491 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:09.014267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.15.18.67:33189)
0929 02:58:09.014922 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:09.014930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:09.014959 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:58:09.015018 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:09.015024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:09.015263 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:09.015404 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:09.016723 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:09.017816 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:09.019264 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:09.020091 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:09.020202 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:09.024616 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:09.024630 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:09.024646 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:09.024685 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:09.027646 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:09.028362 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:09.028372 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:09.028375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:09.028581 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:09.028939 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:09.028944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:09.028946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:09.029337 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:09.029607 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:09.029986 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 02:58:09.030205 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":359,"threads_started":1}
I20260929 02:58:09.031795 28243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:09.117265 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 28120
I20260929 02:58:09.129513 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 27726
I20260929 02:58:09.130836 27788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:09.236851 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 27726
I20260929 02:58:09.253342 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:38719
--webserver_interface=127.15.18.126
--webserver_port=46809
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:38719
--ranger_config_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:46663/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.18.84:46663/kms with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:09.565878 28260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:09.566182 28260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:09.566319 28260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:09.573593 28260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:58:09.573717 28260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:09.573797 28260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:09.573872 28260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:58:09.573938 28260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:58:09.583869 28260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.18.84:46663/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:38719
--ranger_config_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:38719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=46809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:09.585127 28260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:09.586447 28260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:09.597447 28265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:09.597383 28268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:09.597424 28266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:09.598079 28260 server_base.cc:1034] running on GCE node
I20260929 02:58:09.598789 28260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:09.600132 28260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:09.601326 28260 hybrid_clock.cc:648] HybridClock initialized: now 1790650689601283 us; error 39 us; skew 500 ppm
I20260929 02:58:09.601658 28260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:09 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650689, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:09.604555 28260 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:58:09.605964 28260 webserver.cc:460] Webserver started at http://127.15.18.126:46809/ using document root <none> and password file <none>
I20260929 02:58:09.606559 28260 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:09.606684 28260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:58:09 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650689, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:09.667331 28260 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 02:58:09.670204 28276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:09.671499 28260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:58:09.671713 28260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "a1fa5e9b79e84214bd08f2275b5f277a"
format_stamp: "Formatted at 2026-09-29 02:58:05 on dist-test-slave-q5f0"
server_key: "001f26ae512821c25d419d30fadf46cd"
server_key_iv: "07874faa984d910fed3696edfbc5dbb7"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:09.672624 28260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:09.698276 28260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:09.702046 28260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:09.702319 28260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:09.719062 28260 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:38719
I20260929 02:58:09.719081 28327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:38719 every 8 connection(s)
I20260929 02:58:09.720468 28260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:58:09.724265 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 28260
I20260929 02:58:09.725050 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:46663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:46037
--local_ip_for_outbound_sockets=127.15.18.65
--tserver_master_addrs=127.15.18.126:38719
--webserver_port=33397
--webserver_interface=127.15.18.65
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.18.84:46663/kms with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:58:09.726979 28328 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:58:09.730115 28328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: Bootstrap starting.
I20260929 02:58:09.747633 28328 log.cc:824] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:09.756165 28328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:58:09.756650 28328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: Bootstrap complete.
I20260929 02:58:09.765242 28328 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } }
I20260929 02:58:09.766134 28328 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fa5e9b79e84214bd08f2275b5f277a, State: Initialized, Role: FOLLOWER
I20260929 02:58:09.766744 28328 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } }
I20260929 02:58:09.766957 28328 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:58:09.767067 28328 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:58:09.767225 28328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:58:09.771013 28328 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } }
I20260929 02:58:09.771533 28328 leader_election.cc:302] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1fa5e9b79e84214bd08f2275b5f277a; no voters:
I20260929 02:58:09.772123 28328 leader_election.cc:288] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:58:09.772264 28332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:58:09.774230 28328 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:58:09.779166 28332 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [term 2 LEADER]: Becoming Leader. State: Replica: a1fa5e9b79e84214bd08f2275b5f277a, State: Running, Role: LEADER
I20260929 02:58:09.779778 28332 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } }
I20260929 02:58:09.782567 28333 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } } }
I20260929 02:58:09.782939 28333 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:09.783344 28334 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fa5e9b79e84214bd08f2275b5f277a. Latest consensus state: current_term: 2 leader_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fa5e9b79e84214bd08f2275b5f277a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38719 } } }
I20260929 02:58:09.783634 28334 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:09.784916 28342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:58:09.790337 28342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:58:09.791070 28328 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:58:09.792547 28342 catalog_manager.cc:1259] Loaded cluster ID: 265d0703c03c4ffeb104d1188baa62d6
I20260929 02:58:09.792726 28342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:58:09.796769 28342 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:58:09.798260 28342 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: Loaded TSK: 0
I20260929 02:58:09.799883 28342 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:58:10.157764 28330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:10.158320 28330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:10.158540 28330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:10.170011 28330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:10.170252 28330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:10.170478 28330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:58:10.186766 28330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.18.84:46663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:46037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=33397
--tserver_master_addrs=127.15.18.126:38719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:10.188794 28330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:10.190876 28330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:10.215777 28366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:10.220582 28365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:10.221138 28330 server_base.cc:1034] running on GCE node
W20260929 02:58:10.221743 28369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:10.222617 28330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:10.230572 28330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:10.231889 28330 hybrid_clock.cc:648] HybridClock initialized: now 1790650690231813 us; error 71 us; skew 500 ppm
I20260929 02:58:10.232285 28330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:10 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650690, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:10.236486 28330 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:58:10.265897 28330 webserver.cc:460] Webserver started at http://127.15.18.65:33397/ using document root <none> and password file <none>
I20260929 02:58:10.266986 28330 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:10.267175 28330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:58:10 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650690, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:10.289860 28330 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 02:58:10.292768 28376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:10.294256 28330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:58:10.294418 28330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "83945042ea374edc8ee66384ead8ead3"
format_stamp: "Formatted at 2026-09-29 02:58:07 on dist-test-slave-q5f0"
server_key: "f7ae9f7cc43d53e64ebf903aea77e911"
server_key_iv: "6ad880f54dfa9c103a43e6bb3cb37c00"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:10.295428 28330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:10.318205 28330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:10.322350 28330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:10.322609 28330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:10.323623 28330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:10.324959 28330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:10.325052 28330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:10.325157 28330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:10.325224 28330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:10.383149 28330 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:46037
I20260929 02:58:10.383208 28489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:46037 every 8 connection(s)
I20260929 02:58:10.384889 28330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:58:10.392459 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 28330
I20260929 02:58:10.393286 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:46663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:42981
--local_ip_for_outbound_sockets=127.15.18.66
--tserver_master_addrs=127.15.18.126:38719
--webserver_port=42139
--webserver_interface=127.15.18.66
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.18.84:46663/kms with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:58:10 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650690, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:10.414340 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:10.388090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38719 (local address 127.15.18.65:33883)
0929 02:58:10.389617 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:10.389663 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:10.391312 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:10.393480 (+ 2168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:10.393490 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:10.394040 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:10.394848 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:10.394875 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:10.397129 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:10.397135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:10.398986 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:10.399003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:10.400717 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:10.400724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:10.400994 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:10.402336 (+ 1342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:10.402365 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:10.405068 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:10.409253 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:10.409263 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:10.409281 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:10.409860 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:10.410426 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:10.410431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:10.410435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:10.410688 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:10.411849 (+ 1161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:10.411858 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:10.412172 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 02:58:10.412412 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 02:58:10.412997 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"thread_start_us":1106,"threads_started":1}
I20260929 02:58:10.414327 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:10.387955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:33883 (local address 127.15.18.126:38719)
0929 02:58:10.388641 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:10.388659 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:10.389800 (+ 1141us) server_negotiation.cc:407] Connection header received
0929 02:58:10.391910 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:10.391929 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:10.392450 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:10.392859 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:10.395098 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:10.396914 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:10.399180 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:10.400163 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:10.400342 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:10.405398 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:10.405408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:10.405427 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:10.405471 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:10.409043 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:10.410000 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:10.410007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:10.410014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:10.410278 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:10.410819 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:10.410825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:10.410829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:10.411557 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:10.411756 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:10.412644 (+ 888us) server_negotiation.cc:299] Negotiation successful
0929 02:58:10.413018 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":967,"threads_started":1}
I20260929 02:58:10.418387 28490 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38719
I20260929 02:58:10.418751 28490 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:10.419704 28490 heartbeater.cc:511] Master 127.15.18.126:38719 requested a full tablet report, sending...
W20260929 02:58:10.422977 28490 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:38719 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 02:58:10.423380 28490 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:10.423698 28490 heartbeater.cc:511] Master 127.15.18.126:38719 requested a full tablet report, sending...
I20260929 02:58:10.424957 28490 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:10.425293 28490 heartbeater.cc:511] Master 127.15.18.126:38719 requested a full tablet report, sending...
W20260929 02:58:10.426422 28490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:58:10.783455 28494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:10.783938 28494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:10.784101 28494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:10.795377 28494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:10.795552 28494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:10.795748 28494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:58:10.811138 28494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.18.84:46663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:42981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=42139
--tserver_master_addrs=127.15.18.126:38719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:10.812949 28494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:10.814646 28494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:10.830228 28508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:10.830700 28511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:10.830971 28507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:10.830997 28494 server_base.cc:1034] running on GCE node
I20260929 02:58:10.831743 28494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:10.832680 28494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:10.833876 28494 hybrid_clock.cc:648] HybridClock initialized: now 1790650690833809 us; error 58 us; skew 500 ppm
I20260929 02:58:10.834161 28494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:10 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650690, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:10.837582 28494 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:58:10.847872 28494 webserver.cc:460] Webserver started at http://127.15.18.66:42139/ using document root <none> and password file <none>
I20260929 02:58:10.848570 28494 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:10.848704 28494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:58:10 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650690, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:10.868176 28494 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:58:10.871013 28519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:10.872706 28494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:58:10.872898 28494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "97e68dcd328c4ef79520c3181cfce1b5"
format_stamp: "Formatted at 2026-09-29 02:58:07 on dist-test-slave-q5f0"
server_key: "eb10dc223e1ede566a0720730848cdca"
server_key_iv: "854b9027caca1cfbb80995553837417d"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:10.874282 28494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:10.926553 28494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:10.931495 28494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:10.931936 28494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:10.943953 28494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:10.945897 28494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:10.946019 28494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:10.946197 28494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:10.946278 28494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:58:10 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650690, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:11.004372 28494 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:42981
I20260929 02:58:11.004451 28633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:42981 every 8 connection(s)
I20260929 02:58:11.006285 28494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:58:11.013247 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 28494
I20260929 02:58:11.014091 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.18.84:46663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:33189
--local_ip_for_outbound_sockets=127.15.18.67
--tserver_master_addrs=127.15.18.126:38719
--webserver_port=35995
--webserver_interface=127.15.18.67
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.18.84:46663/kms with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:58:11 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650690, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:11.034202 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.010208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38719 (local address 127.15.18.66:52245)
0929 02:58:11.010963 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:11.011016 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:11.013701 (+ 2685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:11.015502 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:11.015540 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:11.016158 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:11.017011 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:11.017038 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.018780 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.018787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:11.020461 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:11.020473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.021301 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.021309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:11.021588 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.022390 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:11.022427 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:11.025538 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:11.029489 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:11.029498 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:11.029518 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:11.030088 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:11.030636 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:11.030640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:11.030643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:11.030870 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:11.031562 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:11.031570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:11.031975 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 02:58:11.032269 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 02:58:11.032804 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":5221,"threads_started":1}
I20260929 02:58:11.035291 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.010784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:52245 (local address 127.15.18.126:38719)
0929 02:58:11.014877 (+ 4093us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:11.014883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:11.014907 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:58:11.014974 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:11.014979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:11.015065 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:11.017223 (+ 2158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:11.017415 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.018582 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.020631 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.021059 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.021177 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.025814 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:11.025820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:11.025824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:11.025875 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:11.029302 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:11.030243 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:11.030249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:11.030253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:11.030511 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:11.030985 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:11.030989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:11.030992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:11.031324 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:11.031457 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:11.034689 (+ 3232us) server_negotiation.cc:299] Negotiation successful
0929 02:58:11.034930 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3959,"thread_start_us":3830,"threads_started":1}
I20260929 02:58:11.037626 28634 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38719
I20260929 02:58:11.037953 28634 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:11.038865 28634 heartbeater.cc:511] Master 127.15.18.126:38719 requested a full tablet report, sending...
I20260929 02:58:11.041041 28293 ts_manager.cc:194] Registered new tserver with Master: 97e68dcd328c4ef79520c3181cfce1b5 (127.15.18.66:42981)
I20260929 02:58:11.043223 28293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:52245
Sep 29 02:58:11 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650690, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:11.426574 28490 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:11.427017 28490 heartbeater.cc:511] Master 127.15.18.126:38719 requested a full tablet report, sending...
I20260929 02:58:11.428372 28293 ts_manager.cc:194] Registered new tserver with Master: 83945042ea374edc8ee66384ead8ead3 (127.15.18.65:46037)
I20260929 02:58:11.429564 28293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:33883
W20260929 02:58:11.472357 28638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:11.472841 28638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:11.473043 28638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:11.484341 28638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:11.484488 28638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:11.484716 28638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:58:11.500108 28638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.18.84:46663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:33189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=35995
--tserver_master_addrs=127.15.18.126:38719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:11.501765 28638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:11.503427 28638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:11.522891 28644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:11.524971 28645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:11.537314 28647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:11.537580 28638 server_base.cc:1034] running on GCE node
I20260929 02:58:11.538528 28638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:11.539824 28638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:11.541124 28638 hybrid_clock.cc:648] HybridClock initialized: now 1790650691541050 us; error 61 us; skew 500 ppm
I20260929 02:58:11.541508 28638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:11 dist-test-slave-q5f0 krb5kdc[27282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650691, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:11.545413 28638 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:58:11.547391 28638 webserver.cc:460] Webserver started at http://127.15.18.67:35995/ using document root <none> and password file <none>
I20260929 02:58:11.548367 28638 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:11.548517 28638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:58:11 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650691, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:11.569024 28638 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:58:11.571750 28655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:11.573089 28638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:58:11.573238 28638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "3d63624ade0743be8d6408317bfa3947"
format_stamp: "Formatted at 2026-09-29 02:58:08 on dist-test-slave-q5f0"
server_key: "9a92293b0798d01b558dee2643eb8339"
server_key_iv: "2383be2102f62fd515f217e63209ea3a"
server_key_version: "kuduclusterkey@0"
I20260929 02:58:11.574200 28638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:11.617221 28638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:11.621112 28638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:11.621357 28638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:11.622380 28638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:11.623819 28638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:11.623915 28638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:11.624019 28638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:11.624089 28638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:58:11 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650690, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for HTTP/127.15.18.84@KRBTEST.COM
I20260929 02:58:11.699139 28638 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:33189
I20260929 02:58:11.699200 28768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:33189 every 8 connection(s)
I20260929 02:58:11.701114 28638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:58:11.707332 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 28638
I20260929 02:58:11.796797 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.763894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.15.18.126:38719)
0929 02:58:11.780811 (+ 16917us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:11.780817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:11.780838 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:58:11.780904 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:11.780909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:11.780986 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:11.781097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:11.782800 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.783632 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.785015 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.785708 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.785837 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.789278 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:11.789285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:11.789289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:11.789333 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:11.792768 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:11.793587 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:11.793591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:11.793595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:11.793744 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:11.794182 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:11.794186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:11.794189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:11.794759 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:11.796033 (+ 1274us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:11.796180 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 02:58:11.796403 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16793,"thread_start_us":16648,"threads_started":1}
Sep 29 02:58:11 dist-test-slave-q5f0 krb5kdc[27282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650691, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:11.807719 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.705636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:60303 (local address 127.15.18.126:38719)
0929 02:58:11.763873 (+ 58237us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:11.763879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:11.763900 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:58:11.775853 (+ 11953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:11.775859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:11.775987 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:11.776079 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:11.786883 (+ 10804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.788180 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.792199 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.792873 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.792952 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.801344 (+ 8392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:11.801350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:11.801354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:11.801403 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:11.804020 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:11.804926 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:11.804931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:11.804934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:11.805070 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:11.805610 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:11.805615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:11.805618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:11.805961 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:11.806090 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:11.807089 (+ 999us) server_negotiation.cc:299] Negotiation successful
0929 02:58:11.807308 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58112,"thread_start_us":58014,"threads_started":1}
I20260929 02:58:11.810146 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.704999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38719 (local address 127.15.18.67:60303)
0929 02:58:11.762268 (+ 57269us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:11.762344 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:11.771303 (+ 8959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:11.777143 (+ 5840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:11.777157 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:11.779578 (+ 2421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:11.785741 (+ 6163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:11.786679 (+ 938us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.788766 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.789305 (+ 539us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:11.791420 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:11.791437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.793695 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.793703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:11.794016 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.795045 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:11.795078 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:11.800113 (+ 5035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:11.804160 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:11.804170 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:11.804189 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:11.804791 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:11.805213 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:11.805217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:11.805221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:11.805451 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:11.806180 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:11.806188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:11.806655 (+ 467us) client_negotiation.cc:769] Sending connection context
0929 02:58:11.808152 (+ 1497us) client_negotiation.cc:240] Negotiation successful
0929 02:58:11.808736 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56900,"thread_start_us":56649,"threads_started":1}
I20260929 02:58:11.816632 28769 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38719
I20260929 02:58:11.816990 28769 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:11.817747 28769 heartbeater.cc:511] Master 127.15.18.126:38719 requested a full tablet report, sending...
I20260929 02:58:11.819320 28293 ts_manager.cc:194] Registered new tserver with Master: 3d63624ade0743be8d6408317bfa3947 (127.15.18.67:33189)
I20260929 02:58:11.820747 28293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:60303
I20260929 02:58:11.821415 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:58:11.848663 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.825224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.15.18.126:38719)
0929 02:58:11.825391 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:11.825398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:11.832057 (+ 6659us) server_negotiation.cc:407] Connection header received
0929 02:58:11.832251 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:11.832258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:11.832347 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:11.832458 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:11.834441 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.835578 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.837643 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.838310 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.838412 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.842008 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:11.842015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:11.842019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:11.842073 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:11.845432 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:11.846256 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:11.846261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:11.846265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:11.846419 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:11.846881 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:11.846886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:11.846890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:11.847254 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:11.847483 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:11.848104 (+ 621us) server_negotiation.cc:299] Negotiation successful
0929 02:58:11.848336 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:58:11.855578 28293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:58:11.866629 28293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:58:11.934015 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.890947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:42981 (local address 127.0.0.1:38366)
0929 02:58:11.891736 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:11.891776 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:11.891968 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:11.926313 (+ 34345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:11.926321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:11.926357 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:11.926739 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:11.926754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.928668 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.928674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:11.931051 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:11.931068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.933005 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.933012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:11.933258 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.933312 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 02:58:11.933422 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 02:58:11.933622 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":3,"thread_start_us":604,"threads_started":1}
I20260929 02:58:11.935025 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.891418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.15.18.66:42981)
0929 02:58:11.925644 (+ 34226us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:11.925654 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:11.925682 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:58:11.925778 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:11.925787 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:11.926034 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:11.926234 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:11.926914 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.928507 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.931229 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.932731 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.934302 (+ 1571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.934510 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:11.934578 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 02:58:11.934742 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34052,"thread_start_us":33956,"threads_started":1}
I20260929 02:58:11.939661 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.894439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.15.18.65:46037)
0929 02:58:11.910793 (+ 16354us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:11.910802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:11.923612 (+ 12810us) server_negotiation.cc:407] Connection header received
0929 02:58:11.923771 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:11.923780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:11.924795 (+ 1015us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:11.925027 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:11.932674 (+ 7647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.934201 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.936444 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.937849 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.937973 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.938194 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:11.939171 (+ 977us) server_negotiation.cc:299] Negotiation successful
0929 02:58:11.939319 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16151,"thread_start_us":16366,"threads_started":1}
I20260929 02:58:11.940138 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.892419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:46037 (local address 127.0.0.1:43406)
0929 02:58:11.922929 (+ 30510us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:11.922954 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:11.923111 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:11.932176 (+ 9065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:11.932181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:11.932204 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:11.932469 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:11.932476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.934633 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.934637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:11.936298 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:11.936310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.938345 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.938351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:11.938517 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.938549 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:58:11.938626 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:58:11.938755 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29665,"thread_start_us":31639,"threads_started":1}
I20260929 02:58:11.939210 28569 tablet_service.cc:1467] Processing CreateTablet for tablet c6b6dfee2db946edbdc897caa0d4599c (DEFAULT_TABLE table=test-table [id=4e34e07a1a9f44088b92a4f9c0fbf862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:11.941941 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.892744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:33189 (local address 127.0.0.1:43044)
0929 02:58:11.924318 (+ 31574us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:11.924337 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:11.924460 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:11.932676 (+ 8216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:11.932680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:11.932704 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:11.933059 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:11.933071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.937570 (+ 4499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.937576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:11.939458 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:11.939472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.941265 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.941271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:11.941429 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.941462 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:58:11.941523 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 02:58:11.941656 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31451,"thread_start_us":29207,"threads_started":1}
I20260929 02:58:11.942040 28569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b6dfee2db946edbdc897caa0d4599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:11.943019 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:11.893345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.15.18.67:33189)
0929 02:58:11.919845 (+ 26500us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:11.919854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:11.931588 (+ 11734us) server_negotiation.cc:407] Connection header received
0929 02:58:11.931704 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:11.931712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:11.931967 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:11.933258 (+ 1291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:11.933548 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.937408 (+ 3860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.939694 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:11.941046 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:11.942257 (+ 1211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:11.942459 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:11.942521 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 02:58:11.942682 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26313,"thread_start_us":26138,"threads_started":1}
I20260929 02:58:11.943933 28425 tablet_service.cc:1467] Processing CreateTablet for tablet c6b6dfee2db946edbdc897caa0d4599c (DEFAULT_TABLE table=test-table [id=4e34e07a1a9f44088b92a4f9c0fbf862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:11.945310 28425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b6dfee2db946edbdc897caa0d4599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:11.950109 28704 tablet_service.cc:1467] Processing CreateTablet for tablet c6b6dfee2db946edbdc897caa0d4599c (DEFAULT_TABLE table=test-table [id=4e34e07a1a9f44088b92a4f9c0fbf862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:11.951457 28704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b6dfee2db946edbdc897caa0d4599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:11.965209 28790 tablet_bootstrap.cc:492] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: Bootstrap starting.
I20260929 02:58:11.972324 28790 tablet_bootstrap.cc:654] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:11.973658 28790 log.cc:824] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:11.991176 28793 tablet_bootstrap.cc:492] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: Bootstrap starting.
I20260929 02:58:11.997795 28792 tablet_bootstrap.cc:492] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: Bootstrap starting.
I20260929 02:58:11.998576 28790 tablet_bootstrap.cc:492] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: No bootstrap required, opened a new log
I20260929 02:58:11.998890 28790 ts_tablet_manager.cc:1397] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.019s
I20260929 02:58:11.999424 28793 tablet_bootstrap.cc:654] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:12.000917 28793 log.cc:824] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:12.002516 28792 tablet_bootstrap.cc:654] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:12.003861 28792 log.cc:824] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:12.010938 28793 tablet_bootstrap.cc:492] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: No bootstrap required, opened a new log
I20260929 02:58:12.011330 28793 ts_tablet_manager.cc:1397] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20260929 02:58:12.014365 28792 tablet_bootstrap.cc:492] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: No bootstrap required, opened a new log
I20260929 02:58:12.014673 28792 ts_tablet_manager.cc:1397] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.013s
I20260929 02:58:12.021903 28793 raft_consensus.cc:348] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 } } peers { permanent_uuid: "3d63624ade0743be8d6408317bfa3947" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33189 } } peers { permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 } }
I20260929 02:58:12.022686 28793 raft_consensus.cc:374] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:12.022853 28793 raft_consensus.cc:729] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83945042ea374edc8ee66384ead8ead3, State: Initialized, Role: FOLLOWER
I20260929 02:58:12.023715 28793 consensus_queue.cc:260] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 } } peers { permanent_uuid: "3d63624ade0743be8d6408317bfa3947" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33189 } } peers { permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 } }
I20260929 02:58:12.028303 28790 raft_consensus.cc:348] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 } } peers { permanent_uuid: "3d63624ade0743be8d6408317bfa3947" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33189 } } peers { permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 } }
I20260929 02:58:12.028750 28790 raft_consensus.cc:374] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:12.028869 28790 raft_consensus.cc:729] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e68dcd328c4ef79520c3181cfce1b5, State: Initialized, Role: FOLLOWER
I20260929 02:58:12.029776 28790 consensus_queue.cc:260] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 } } peers { permanent_uuid: "3d63624ade0743be8d6408317bfa3947" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33189 } } peers { permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 } }
I20260929 02:58:12.036468 28792 raft_consensus.cc:348] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 } } peers { permanent_uuid: "3d63624ade0743be8d6408317bfa3947" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33189 } } peers { permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 } }
I20260929 02:58:12.036963 28792 raft_consensus.cc:374] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:12.037072 28792 raft_consensus.cc:729] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d63624ade0743be8d6408317bfa3947, State: Initialized, Role: FOLLOWER
I20260929 02:58:12.037685 28634 heartbeater.cc:503] Master 127.15.18.126:38719 was elected leader, sending a full tablet report...
I20260929 02:58:12.037729 28792 consensus_queue.cc:260] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 } } peers { permanent_uuid: "3d63624ade0743be8d6408317bfa3947" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33189 } } peers { permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 } }
I20260929 02:58:12.038597 28790 ts_tablet_manager.cc:1428] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: Time spent starting tablet: real 0.040s user 0.006s sys 0.015s
I20260929 02:58:12.046429 28769 heartbeater.cc:503] Master 127.15.18.126:38719 was elected leader, sending a full tablet report...
I20260929 02:58:12.046662 28792 ts_tablet_manager.cc:1428] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: Time spent starting tablet: real 0.032s user 0.011s sys 0.010s
I20260929 02:58:12.052222 28490 heartbeater.cc:503] Master 127.15.18.126:38719 was elected leader, sending a full tablet report...
I20260929 02:58:12.053061 28793 ts_tablet_manager.cc:1428] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: Time spent starting tablet: real 0.042s user 0.023s sys 0.014s
W20260929 02:58:12.137588 28491 tablet.cc:2369] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:58:12.260149 28635 tablet.cc:2369] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:58:12.279649 28770 tablet.cc:2369] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:58:12.315824 28798 raft_consensus.cc:482] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:58:12.316092 28798 raft_consensus.cc:504] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 } } peers { permanent_uuid: "3d63624ade0743be8d6408317bfa3947" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33189 } } peers { permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 } }
I20260929 02:58:12.317723 28798 leader_election.cc:288] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97e68dcd328c4ef79520c3181cfce1b5 (127.15.18.66:42981), 83945042ea374edc8ee66384ead8ead3 (127.15.18.65:46037)
I20260929 02:58:12.324368 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:12.318076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:52363 (local address 127.15.18.66:42981)
0929 02:58:12.318300 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:12.318306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:12.318489 (+ 183us) server_negotiation.cc:407] Connection header received
0929 02:58:12.318723 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:12.318729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:12.318812 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:12.318900 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:12.319442 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.320366 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.322429 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.323398 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.323573 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:12.323645 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:12.323954 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 02:58:12.324098 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:58:12.324410 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:12.317935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:42981 (local address 127.15.18.67:52363)
0929 02:58:12.318377 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:12.318397 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:12.318553 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:12.318982 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:12.318988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:12.319018 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:12.319287 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:12.319300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.320517 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.320524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:12.322246 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:12.322265 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.323593 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.323599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:12.323784 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:12.323821 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:58:12.323926 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 02:58:12.324061 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260929 02:58:12.324962 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:12.318700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:35003 (local address 127.15.18.65:46037)
0929 02:58:12.318906 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:12.318911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:12.319117 (+ 206us) server_negotiation.cc:407] Connection header received
0929 02:58:12.319237 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:12.319242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:12.319315 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:12.319407 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:12.320053 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.320937 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.322955 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.323953 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.324090 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:12.324155 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:12.324678 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 02:58:12.324790 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:58:12.325446 28807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:12.318508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:46037 (local address 127.15.18.67:35003)
0929 02:58:12.319028 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:12.319043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:12.319140 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:12.319431 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:12.319434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:12.319455 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:12.319869 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:12.319881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.321118 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.321122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:12.322777 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:12.322792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.324296 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.324302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:12.324486 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:12.324518 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:58:12.325121 (+ 603us) client_negotiation.cc:240] Negotiation successful
0929 02:58:12.325228 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":252,"threads_started":1}
I20260929 02:58:12.325474 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b6dfee2db946edbdc897caa0d4599c" candidate_uuid: "3d63624ade0743be8d6408317bfa3947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e68dcd328c4ef79520c3181cfce1b5" is_pre_election: true
I20260929 02:58:12.326038 28589 raft_consensus.cc:2454] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d63624ade0743be8d6408317bfa3947 in term 0.
I20260929 02:58:12.326383 28445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b6dfee2db946edbdc897caa0d4599c" candidate_uuid: "3d63624ade0743be8d6408317bfa3947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83945042ea374edc8ee66384ead8ead3" is_pre_election: true
I20260929 02:58:12.326730 28658 leader_election.cc:302] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d63624ade0743be8d6408317bfa3947, 97e68dcd328c4ef79520c3181cfce1b5; no voters:
I20260929 02:58:12.326844 28445 raft_consensus.cc:2454] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d63624ade0743be8d6408317bfa3947 in term 0.
I20260929 02:58:12.327071 28798 raft_consensus.cc:2781] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:58:12.327186 28798 raft_consensus.cc:482] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:58:12.327291 28798 raft_consensus.cc:3037] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:12.329358 28798 raft_consensus.cc:504] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 } } peers { permanent_uuid: "3d63624ade0743be8d6408317bfa3947" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33189 } } peers { permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 } }
I20260929 02:58:12.329898 28798 leader_election.cc:288] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [CANDIDATE]: Term 1 election: Requested vote from peers 97e68dcd328c4ef79520c3181cfce1b5 (127.15.18.66:42981), 83945042ea374edc8ee66384ead8ead3 (127.15.18.65:46037)
I20260929 02:58:12.330420 28445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b6dfee2db946edbdc897caa0d4599c" candidate_uuid: "3d63624ade0743be8d6408317bfa3947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83945042ea374edc8ee66384ead8ead3"
I20260929 02:58:12.330376 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b6dfee2db946edbdc897caa0d4599c" candidate_uuid: "3d63624ade0743be8d6408317bfa3947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e68dcd328c4ef79520c3181cfce1b5"
I20260929 02:58:12.330612 28445 raft_consensus.cc:3037] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:12.330612 28589 raft_consensus.cc:3037] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:12.333664 28445 raft_consensus.cc:2454] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d63624ade0743be8d6408317bfa3947 in term 1.
I20260929 02:58:12.333664 28589 raft_consensus.cc:2454] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d63624ade0743be8d6408317bfa3947 in term 1.
I20260929 02:58:12.334026 28658 leader_election.cc:302] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d63624ade0743be8d6408317bfa3947, 97e68dcd328c4ef79520c3181cfce1b5; no voters:
I20260929 02:58:12.334307 28798 raft_consensus.cc:2781] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:58:12.334692 28798 raft_consensus.cc:686] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 1 LEADER]: Becoming Leader. State: Replica: 3d63624ade0743be8d6408317bfa3947, State: Running, Role: LEADER
I20260929 02:58:12.335132 28798 consensus_queue.cc:237] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 } } peers { permanent_uuid: "3d63624ade0743be8d6408317bfa3947" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33189 } } peers { permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 } }
I20260929 02:58:12.339401 28293 catalog_manager.cc:5495] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d63624ade0743be8d6408317bfa3947 (127.15.18.67). New cstate: current_term: 1 leader_uuid: "3d63624ade0743be8d6408317bfa3947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d63624ade0743be8d6408317bfa3947" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20260929 02:58:12.354997 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:12.349914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.15.18.67:33189)
0929 02:58:12.350094 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:12.350098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:12.350207 (+ 109us) server_negotiation.cc:407] Connection header received
0929 02:58:12.350389 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:12.350393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:12.350465 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:12.350577 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:58:12.350961 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.351733 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.353042 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.353441 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.353529 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:12.354006 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:58:12.354495 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:58:12.354597 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:12.354692 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 02:58:12.354755 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:58:12.362541 28445 raft_consensus.cc:1261] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3 [term 1 FOLLOWER]: Refusing update from remote peer 3d63624ade0743be8d6408317bfa3947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:58:12.362557 28589 raft_consensus.cc:1261] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5 [term 1 FOLLOWER]: Refusing update from remote peer 3d63624ade0743be8d6408317bfa3947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:58:12.363405 28808 consensus_queue.cc:1035] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83945042ea374edc8ee66384ead8ead3" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:58:12.363794 28798 consensus_queue.cc:1035] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e68dcd328c4ef79520c3181cfce1b5" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:58:12.375571 28812 mvcc.cc:205] Tried to move back new op lower bound from 7334505235907825664 to 7334505235808448512. Current Snapshot: MvccSnapshot[applied={T|T < 7334505235907825664}]
I20260929 02:58:12.376085 28813 mvcc.cc:205] Tried to move back new op lower bound from 7334505235907825664 to 7334505235808448512. Current Snapshot: MvccSnapshot[applied={T|T < 7334505235907825664}]
I20260929 02:58:12.389256 28293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:58:12.389526 28293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:58:12.393391 28293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1fa5e9b79e84214bd08f2275b5f277a: Sending DeleteTablet for 3 replicas of tablet c6b6dfee2db946edbdc897caa0d4599c
I20260929 02:58:12.394536 28704 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b6dfee2db946edbdc897caa0d4599c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:12 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:43044
I20260929 02:58:12.394553 28569 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b6dfee2db946edbdc897caa0d4599c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:12 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:38366
I20260929 02:58:12.400413 28425 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b6dfee2db946edbdc897caa0d4599c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:12 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:43406
I20260929 02:58:12.403247 28821 tablet_replica.cc:330] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: stopping tablet replica
I20260929 02:58:12.403805 28821 raft_consensus.cc:2229] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:58:12.404389 28821 raft_consensus.cc:2258] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:12.405189 28820 tablet_replica.cc:330] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: stopping tablet replica
I20260929 02:58:12.405608 28820 raft_consensus.cc:2229] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:58:12.406260 28820 raft_consensus.cc:2258] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:12.406797 28821 ts_tablet_manager.cc:1905] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:12.408308 28820 ts_tablet_manager.cc:1905] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:12.412153 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 28330
I20260929 02:58:12.413918 28821 ts_tablet_manager.cc:1918] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:12.414175 28821 log.cc:1196] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/c6b6dfee2db946edbdc897caa0d4599c
I20260929 02:58:12.414695 28821 ts_tablet_manager.cc:1939] T c6b6dfee2db946edbdc897caa0d4599c P 97e68dcd328c4ef79520c3181cfce1b5: Deleting consensus metadata
I20260929 02:58:12.416703 28279 catalog_manager.cc:4841] TS 97e68dcd328c4ef79520c3181cfce1b5 (127.15.18.66:42981): tablet c6b6dfee2db946edbdc897caa0d4599c (table test-table [id=4e34e07a1a9f44088b92a4f9c0fbf862]) successfully deleted
I20260929 02:58:12.422703 28822 tablet_replica.cc:330] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: stopping tablet replica
I20260929 02:58:12.422766 28820 ts_tablet_manager.cc:1918] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:12.422968 28820 log.cc:1196] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/c6b6dfee2db946edbdc897caa0d4599c
I20260929 02:58:12.423565 28820 ts_tablet_manager.cc:1939] T c6b6dfee2db946edbdc897caa0d4599c P 3d63624ade0743be8d6408317bfa3947: Deleting consensus metadata
I20260929 02:58:12.423436 28822 raft_consensus.cc:2229] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:58:12.424985 28822 raft_consensus.cc:2258] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:12.425271 28280 catalog_manager.cc:4841] TS 3d63624ade0743be8d6408317bfa3947 (127.15.18.67:33189): tablet c6b6dfee2db946edbdc897caa0d4599c (table test-table [id=4e34e07a1a9f44088b92a4f9c0fbf862]) successfully deleted
I20260929 02:58:12.426671 28822 ts_tablet_manager.cc:1905] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:12.431941 28822 ts_tablet_manager.cc:1918] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:12.432113 28822 log.cc:1196] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/c6b6dfee2db946edbdc897caa0d4599c
I20260929 02:58:12.432571 28822 ts_tablet_manager.cc:1939] T c6b6dfee2db946edbdc897caa0d4599c P 83945042ea374edc8ee66384ead8ead3: Deleting consensus metadata
I20260929 02:58:12.434190 28279 catalog_manager.cc:4841] TS 83945042ea374edc8ee66384ead8ead3 (127.15.18.65:46037): tablet c6b6dfee2db946edbdc897caa0d4599c (table test-table [id=4e34e07a1a9f44088b92a4f9c0fbf862]) successfully deleted
I20260929 02:58:12.439268 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:12.413005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.15.18.65:46037)
0929 02:58:12.413198 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:12.413204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:12.424071 (+ 10867us) server_negotiation.cc:407] Connection header received
0929 02:58:12.424321 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:12.424326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:12.424407 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:12.424516 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:12.426534 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.427643 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.429425 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.430074 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.430205 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:12.433591 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:12.433602 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:12.433619 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:12.433661 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:12.436504 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:12.437326 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:12.437333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:12.437336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:12.437471 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:12.437882 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:12.437887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:12.437889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:12.438256 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:12.438485 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:12.438832 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 02:58:12.439003 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:58:12.439898 28485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:12.589072 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 28330
I20260929 02:58:12.609019 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 28494
I20260929 02:58:12.624155 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:12.609916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38380 (local address 127.15.18.66:42981)
0929 02:58:12.610075 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:12.610079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:12.610097 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:58:12.610285 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:12.610290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:12.610357 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:12.610439 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:12.612177 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.612888 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.614699 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.615147 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.615251 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:12.618408 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:12.618421 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:12.618437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:12.618476 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:12.621207 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:12.622075 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:12.622082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:12.622084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:12.622235 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:12.622635 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:12.622639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:12.622641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:12.622993 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:12.623232 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:12.623650 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 02:58:12.623822 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:58:12.624840 28629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:12.723688 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 28494
I20260929 02:58:12.743767 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 28638
I20260929 02:58:12.759582 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:12.744502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.15.18.67:33189)
0929 02:58:12.744696 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:12.744702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:12.744729 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:58:12.744855 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:12.744861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:12.744968 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:12.745083 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:12.746289 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.747460 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.748798 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:12.749483 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:12.749621 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:12.752119 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:12.752136 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:12.752157 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:12.752212 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:12.756559 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:12.757337 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:12.757347 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:12.757350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:12.757536 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:12.757863 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:12.757868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:12.757871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:12.758373 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:12.758686 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:12.758917 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 02:58:12.759150 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:58:12.760290 28764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:12.880159 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 28638
I20260929 02:58:12.895399 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 28260
I20260929 02:58:12.896508 28323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:13.007774 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 28260
I20260929 02:58:13.028179 15433 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 02:58:13.363787 15433 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 02:58:13.363984 15433 mini_ranger.cc:66] Stopping Ranger...
I20260929 02:58:13.709473 15433 mini_ranger.cc:68] Stopped Ranger
2026-09-29 02:58:13.709 UTC [27312] LOG: received smart shutdown request
2026-09-29 02:58:13.712 UTC [27312] LOG: background worker "logical replication launcher" (PID 27321) exited with exit code 1
2026-09-29 02:58:13.712 UTC [27316] LOG: shutting down
2026-09-29 02:58:13.881 UTC [27312] LOG: database system is shut down
2026-09-29T02:58:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52442 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:58:14 dist-test-slave-q5f0 krb5kdc[28836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:58:14 dist-test-slave-q5f0 krb5kdc[28836](info): set up 2 sockets
Sep 29 02:58:14 dist-test-slave-q5f0 krb5kdc[28836](info): commencing operation
krb5kdc: starting...
W20260929 02:58:15.000377 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:58:15 dist-test-slave-q5f0 krb5kdc[28836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:58:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:58:15.211776 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:33619
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:33619,127.15.18.125:41567,127.15.18.124:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:15.531370 28849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:15.531697 28849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:15.531798 28849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:15.539012 28849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:58:15.539134 28849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:15.539179 28849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:15.539230 28849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:58:15.539271 28849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:58:15.549085 28849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:33619,127.15.18.125:41567,127.15.18.124:46821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:15.550374 28849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:15.551616 28849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:15.562654 28854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:15.562604 28857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:15.562680 28855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:15.563715 28849 server_base.cc:1034] running on GCE node
I20260929 02:58:15.564404 28849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:15.565734 28849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:15.566931 28849 hybrid_clock.cc:648] HybridClock initialized: now 1790650695566892 us; error 31 us; skew 500 ppm
I20260929 02:58:15.567274 28849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:15 dist-test-slave-q5f0 krb5kdc[28836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:15.570264 28849 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:58:15.571676 28849 webserver.cc:460] Webserver started at http://127.15.18.126:39341/ using document root <none> and password file <none>
I20260929 02:58:15.572304 28849 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:15.572394 28849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:15.572623 28849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:15.575433 28849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "c6b244072da044ffb68da8886fa4f43e"
format_stamp: "Formatted at 2026-09-29 02:58:15 on dist-test-slave-q5f0"
server_key: "3e7536038d9d875f695ca1e1cc85f82e"
server_key_iv: "67e26f7fd8a0641e7a89a46d39abfc79"
server_key_version: "encryptionkey@0"
I20260929 02:58:15.576217 28849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "c6b244072da044ffb68da8886fa4f43e"
format_stamp: "Formatted at 2026-09-29 02:58:15 on dist-test-slave-q5f0"
server_key: "3e7536038d9d875f695ca1e1cc85f82e"
server_key_iv: "67e26f7fd8a0641e7a89a46d39abfc79"
server_key_version: "encryptionkey@0"
I20260929 02:58:15.581048 28849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:58:15.584414 28864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:15.585747 28849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:58:15.585891 28849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "c6b244072da044ffb68da8886fa4f43e"
format_stamp: "Formatted at 2026-09-29 02:58:15 on dist-test-slave-q5f0"
server_key: "3e7536038d9d875f695ca1e1cc85f82e"
server_key_iv: "67e26f7fd8a0641e7a89a46d39abfc79"
server_key_version: "encryptionkey@0"
I20260929 02:58:15.586062 28849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:15.598084 28849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:15.602110 28849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:15.602373 28849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:15.620245 28849 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:33619
I20260929 02:58:15.620308 28915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:33619 every 8 connection(s)
I20260929 02:58:15.621709 28849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:58:15.622332 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 28849
I20260929 02:58:15.622565 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:58:15.623334 15433 external_mini_cluster.cc:1428] Setting key 145f1c29a7b7ad7543768bcbe6afd204
I20260929 02:58:15.625770 28916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:15.633670 28916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.18.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.125@KRBTEST.COM" created.
Sep 29 02:58:15 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:15.658182 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:15.637376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:33619 (local address 127.0.0.1:45318)
0929 02:58:15.638636 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:15.638697 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:15.641223 (+ 2526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:15.642232 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:15.642241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:15.643010 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:15.643484 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:15.643537 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:15.645128 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:15.645132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:15.646479 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:15.646492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:15.647733 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:15.647738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:15.647862 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:15.648463 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:15.648484 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:15.651058 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:15.654515 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:15.654524 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:15.654533 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:15.654930 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:15.655321 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:15.655324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:15.655326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:15.655464 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:15.656358 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:15.656363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:15.656541 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 02:58:15.656760 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 02:58:15.657126 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"thread_start_us":762,"threads_started":1}
I20260929 02:58:15.658209 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:15.637376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.15.18.126:33619)
0929 02:58:15.638347 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:15.638374 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:15.638834 (+ 460us) server_negotiation.cc:407] Connection header received
0929 02:58:15.641689 (+ 2855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:15.641696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:15.641959 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:15.642137 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:15.643758 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:15.644972 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:15.646655 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:15.647474 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:15.647702 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:15.651279 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:15.651288 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:15.651304 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:15.651335 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:15.654399 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:15.655046 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:15.655054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:15.655057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:15.655220 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:15.655613 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:15.655617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:15.655620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:15.656240 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:15.656432 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:15.656921 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 02:58:15.657283 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":706,"spinlock_wait_cycles":16768,"thread_start_us":475,"threads_started":1}
I20260929 02:58:15.661074 28916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:58:15.662297 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:15.661837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41610)
0929 02:58:15.662029 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:15.662156 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 02:58:15.662629 28866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
I20260929 02:58:15.664234 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:15.663811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41612)
0929 02:58:15.663986 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:15.664082 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:58:15.664438 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:41567: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab.
Entry for principal kudu/127.15.18.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab.
I20260929 02:58:15.672479 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.125:41567
--webserver_interface=127.15.18.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab
--principal=kudu/127.15.18.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:33619,127.15.18.125:41567,127.15.18.124:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:58:15.713835 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } attempt: 1
I20260929 02:58:15.715181 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:15.714605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41620)
0929 02:58:15.714855 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:15.714988 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 02:58:15.716305 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:15.715858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41634)
0929 02:58:15.716036 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:15.716132 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 02:58:15.716547 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:41567: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
I20260929 02:58:15.784832 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } attempt: 2
I20260929 02:58:15.786139 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:15.785575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41638)
0929 02:58:15.785838 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:15.785950 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 02:58:15.787238 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:15.786784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41646)
0929 02:58:15.786951 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:15.787070 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 02:58:15.787452 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:41567: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
I20260929 02:58:15.878746 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } attempt: 3
I20260929 02:58:15.880018 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:15.879482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41662)
0929 02:58:15.879733 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:15.879842 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 02:58:15.881028 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:15.880605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41676)
0929 02:58:15.880757 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:15.880854 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:58:15.881203 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:41567: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
W20260929 02:58:15.988191 28923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:15.988513 28923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:15.988652 28923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:15.996097 28923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:58:15.996218 28923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:15.996299 28923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:15.996382 28923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:58:15.996446 28923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:58:16.007745 28923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:33619,127.15.18.125:41567,127.15.18.124:46821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab
--principal=kudu/127.15.18.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.125:41567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.18.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:16.009138 28923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:16.010524 28923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:16.022544 28929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:16.022650 28928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:16.022686 28931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:16.023448 28923 server_base.cc:1034] running on GCE node
I20260929 02:58:16.024259 28923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:16.024437 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } attempt: 4
I20260929 02:58:16.025563 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.025021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41678)
0929 02:58:16.025320 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.025422 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260929 02:58:16.025876 28923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:16.026616 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.026217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41694)
0929 02:58:16.026361 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.026464 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:58:16.026834 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:41567: Network error: Client connection negotiation failed: client connection to 127.15.18.125:41567: connect: Connection refused (error 111)
I20260929 02:58:16.027074 28923 hybrid_clock.cc:648] HybridClock initialized: now 1790650696027023 us; error 46 us; skew 500 ppm
I20260929 02:58:16.027397 28923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650696, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:16.033538 28923 init.cc:377] Logged in from keytab as kudu/127.15.18.125@KRBTEST.COM (short username kudu)
I20260929 02:58:16.034889 28923 webserver.cc:460] Webserver started at http://127.15.18.125:45423/ using document root <none> and password file <none>
I20260929 02:58:16.035562 28923 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:16.035678 28923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:16.035915 28923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:16.038740 28923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/data/instance:
uuid: "bc97cf31d4a1475d85887f50f628e7ee"
format_stamp: "Formatted at 2026-09-29 02:58:16 on dist-test-slave-q5f0"
server_key: "8162ea871a036288dc656350ab523d4c"
server_key_iv: "f0474380157a45c47241716ee3c03573"
server_key_version: "encryptionkey@0"
I20260929 02:58:16.039459 28923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/wal/instance:
uuid: "bc97cf31d4a1475d85887f50f628e7ee"
format_stamp: "Formatted at 2026-09-29 02:58:16 on dist-test-slave-q5f0"
server_key: "8162ea871a036288dc656350ab523d4c"
server_key_iv: "f0474380157a45c47241716ee3c03573"
server_key_version: "encryptionkey@0"
I20260929 02:58:16.044456 28923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:58:16.047693 28938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:16.048967 28923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:58:16.049096 28923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/wal
uuid: "bc97cf31d4a1475d85887f50f628e7ee"
format_stamp: "Formatted at 2026-09-29 02:58:16 on dist-test-slave-q5f0"
server_key: "8162ea871a036288dc656350ab523d4c"
server_key_iv: "f0474380157a45c47241716ee3c03573"
server_key_version: "encryptionkey@0"
I20260929 02:58:16.049268 28923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:16.066469 28923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:16.070340 28923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:16.070564 28923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:16.087380 28923 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.125:41567
I20260929 02:58:16.087410 28989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.125:41567 every 8 connection(s)
I20260929 02:58:16.088685 28923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
I20260929 02:58:16.092366 28990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:16.092511 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 28923
I20260929 02:58:16.092689 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-1/wal/instance
I20260929 02:58:16.093087 15433 external_mini_cluster.cc:1428] Setting key ab48c0ad302948a2f64f497a81781766
I20260929 02:58:16.098558 28990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.18.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.124@KRBTEST.COM" created.
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650696, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:16.120512 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.101355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.15.18.126:33619)
0929 02:58:16.101568 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.101574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.102771 (+ 1197us) server_negotiation.cc:407] Connection header received
0929 02:58:16.104645 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.104650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.104728 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.104838 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.106276 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.107306 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.108921 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.109585 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.109723 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.114434 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.114440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.114443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.114479 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.117087 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.117956 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.117961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.117963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.118105 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.118560 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.118564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.118565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.118895 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.119038 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.119949 (+ 911us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.120193 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:58:16.121417 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.101838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:33619 (local address 127.0.0.1:45326)
0929 02:58:16.102591 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.102645 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:16.104362 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:16.105019 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:16.105028 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:16.105519 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:16.106102 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.106119 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.107487 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.107491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.108774 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.108787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.109789 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.109794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.109983 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.110656 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:16.110678 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:16.114188 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:16.117248 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.117258 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.117276 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.117836 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.118199 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.118204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.118207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.118441 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.119015 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:16.119024 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:16.119561 (+ 537us) client_negotiation.cc:769] Sending connection context
0929 02:58:16.119800 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 02:58:16.120197 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":307,"threads_started":1}
I20260929 02:58:16.123109 28990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650696, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
I20260929 02:58:16.137152 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.123938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41704)
0929 02:58:16.124148 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.124167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:16.124253 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:16.125244 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:16.125249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:16.125678 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:16.125806 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.125814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.127181 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.127185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.128042 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.128053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.128823 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.128829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.129013 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.129541 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:16.129557 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:16.131651 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:16.134634 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.134638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.134641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.135065 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.135466 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.135470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.135473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.135678 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.136294 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:16.136298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:16.136458 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 02:58:16.136616 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 02:58:16.136856 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:58:16.137580 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.124216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.15.18.125:41567)
0929 02:58:16.124849 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.124855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.124878 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:58:16.124918 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.124924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.125095 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.125234 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.125945 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.127055 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.128168 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.128693 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.128807 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.131855 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.131865 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.131880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.131913 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.134499 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.135191 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.135198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.135200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.135312 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.135777 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.135781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.135783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.136165 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.136330 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.136832 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.137042 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":356,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab.
Entry for principal kudu/127.15.18.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab.
I20260929 02:58:16.139930 28990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:58:16.141073 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.140619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43832)
0929 02:58:16.140819 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.140923 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:58:16.141093 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.124:46821
--webserver_interface=127.15.18.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab
--principal=kudu/127.15.18.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:33619,127.15.18.125:41567,127.15.18.124:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:16.141330 28939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
I20260929 02:58:16.142798 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.142487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43842)
0929 02:58:16.142622 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.142700 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:58:16.142992 28990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:46821: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
I20260929 02:58:16.192325 28990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } attempt: 1
I20260929 02:58:16.193637 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.193101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43850)
0929 02:58:16.193313 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.193444 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 02:58:16.194648 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.194272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43858)
0929 02:58:16.194422 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.194521 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:58:16.194888 28990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:46821: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
I20260929 02:58:16.263129 28990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } attempt: 2
I20260929 02:58:16.264401 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.263794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43860)
0929 02:58:16.264055 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.264187 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:58:16.265436 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.265051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43866)
0929 02:58:16.265210 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.265295 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 02:58:16.265697 28990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:46821: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
I20260929 02:58:16.326071 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } attempt: 5
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
I20260929 02:58:16.340014 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.326854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41714)
0929 02:58:16.327169 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.327185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:16.327316 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:16.327698 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:16.327703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:16.328106 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:16.328289 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.328299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.329303 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.329308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.330176 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.330188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.330880 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.330886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.331063 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.331656 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:16.331674 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:16.333823 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:16.337659 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.337663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.337667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.338052 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.338452 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.338455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.338458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.338596 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.339216 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:16.339220 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:16.339366 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 02:58:16.339488 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 02:58:16.339763 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260929 02:58:16.340183 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.326889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.15.18.125:41567)
0929 02:58:16.327092 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.327099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.327259 (+ 160us) server_negotiation.cc:407] Connection header received
0929 02:58:16.327435 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.327441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.327549 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.327642 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.328455 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.329170 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.330321 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.330743 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.330860 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.334040 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.334047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.334052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.334093 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.337454 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.338173 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.338179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.338183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.338336 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.338696 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.338701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.338705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.339092 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.339256 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.339651 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.339878 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:58:16.341313 28916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:58:16.342329 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.341930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43874)
0929 02:58:16.342093 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.342190 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:58:16.343205 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.342886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43884)
0929 02:58:16.343022 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.343087 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 02:58:16.343396 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:46821: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
I20260929 02:58:16.356912 28990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } attempt: 3
I20260929 02:58:16.358188 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.357563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43896)
0929 02:58:16.357834 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.357960 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 02:58:16.359377 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.358931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43904)
0929 02:58:16.359171 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.359257 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260929 02:58:16.359660 28990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:46821: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
I20260929 02:58:16.394636 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } attempt: 1
I20260929 02:58:16.395967 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.395373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43914)
0929 02:58:16.395668 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.395769 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 02:58:16.396997 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.396646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43922)
0929 02:58:16.396794 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.396880 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:58:16.397217 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:46821: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
W20260929 02:58:16.462836 28997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:16.463146 28997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:16.463279 28997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 02:58:16.465437 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } attempt: 2
I20260929 02:58:16.466569 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.466103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43930)
0929 02:58:16.466294 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.466404 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 02:58:16.467479 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.467160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43942)
0929 02:58:16.467300 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.467367 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 02:58:16.467708 28916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:46821: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
W20260929 02:58:16.470996 28997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:58:16.471134 28997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:16.471211 28997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:16.471287 28997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:58:16.471346 28997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:58:16.481285 28997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:33619,127.15.18.125:41567,127.15.18.124:46821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab
--principal=kudu/127.15.18.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.124:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.18.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:16.482569 28997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:16.483927 28997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:16.495414 29005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:16.495473 29002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:16.495442 29003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:16.496646 28997 server_base.cc:1034] running on GCE node
I20260929 02:58:16.497422 28997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:16.498795 28997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:16.499987 28997 hybrid_clock.cc:648] HybridClock initialized: now 1790650696499933 us; error 52 us; skew 500 ppm
I20260929 02:58:16.500317 28997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650696, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:16.502858 28990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } attempt: 4
I20260929 02:58:16.503314 28997 init.cc:377] Logged in from keytab as kudu/127.15.18.124@KRBTEST.COM (short username kudu)
I20260929 02:58:16.503930 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.503425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43954)
0929 02:58:16.503659 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.503745 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 02:58:16.504803 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.504491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43956)
0929 02:58:16.504625 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.504703 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 02:58:16.504827 28997 webserver.cc:460] Webserver started at http://127.15.18.124:41481/ using document root <none> and password file <none>
W20260929 02:58:16.504977 28990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:46821: Network error: Client connection negotiation failed: client connection to 127.15.18.124:46821: connect: Connection refused (error 111)
I20260929 02:58:16.505507 28997 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:16.505614 28997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:16.505890 28997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:16.508754 28997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/data/instance:
uuid: "ecc164ccddd3495b87cb71d88158b54d"
format_stamp: "Formatted at 2026-09-29 02:58:16 on dist-test-slave-q5f0"
server_key: "916812e1162634213f525408c56b0f67"
server_key_iv: "4b7dedf0f37839f75509e2a0fa9e5e66"
server_key_version: "encryptionkey@0"
I20260929 02:58:16.509498 28997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/wal/instance:
uuid: "ecc164ccddd3495b87cb71d88158b54d"
format_stamp: "Formatted at 2026-09-29 02:58:16 on dist-test-slave-q5f0"
server_key: "916812e1162634213f525408c56b0f67"
server_key_iv: "4b7dedf0f37839f75509e2a0fa9e5e66"
server_key_version: "encryptionkey@0"
I20260929 02:58:16.514330 28997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:58:16.517637 29012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:16.519026 28997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:58:16.519202 28997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/wal
uuid: "ecc164ccddd3495b87cb71d88158b54d"
format_stamp: "Formatted at 2026-09-29 02:58:16 on dist-test-slave-q5f0"
server_key: "916812e1162634213f525408c56b0f67"
server_key_iv: "4b7dedf0f37839f75509e2a0fa9e5e66"
server_key_version: "encryptionkey@0"
I20260929 02:58:16.519381 28997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:16.549170 28997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:16.553470 28997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:16.553726 28997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:16.570722 28997 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.124:46821
I20260929 02:58:16.570744 29063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.124:46821 every 8 connection(s)
I20260929 02:58:16.572074 28997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
I20260929 02:58:16.573909 28916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } attempt: 3
I20260929 02:58:16.575958 29064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:16.581566 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 28997
I20260929 02:58:16.581748 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790650496826590-15433-0/minicluster-data/master-2/wal/instance
I20260929 02:58:16.582171 15433 external_mini_cluster.cc:1428] Setting key bb4238cb3c0c1e0b15787e22ef41254d
I20260929 02:58:16.584131 29064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
I20260929 02:58:16.596108 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.574534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43972)
0929 02:58:16.574770 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.574791 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:16.574923 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:16.578581 (+ 3658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:16.578587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:16.579082 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:16.579293 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.579307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.580973 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.580979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.582506 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.582524 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.583911 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.583919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.584120 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.584816 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:16.584840 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:16.588295 (+ 3455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:16.593213 (+ 4918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.593219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.593223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.593600 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.594014 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.594019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.594023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.594159 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.595045 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:16.595050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:16.595240 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 02:58:16.595418 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 02:58:16.595731 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:58:16.597589 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.575409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.15.18.124:46821)
0929 02:58:16.576236 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.576246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.576283 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:58:16.577820 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.577841 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.578220 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.578535 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.579464 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.580814 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.582662 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.583718 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.584549 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.588694 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.588709 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.588733 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.588782 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.593081 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.593723 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.593730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.593736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.593896 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.594260 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.594263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.594265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.594898 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.595666 (+ 768us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.595955 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.596450 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":360,"threads_started":1}
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650696, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
I20260929 02:58:16.604262 28916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e: Bootstrap starting.
I20260929 02:58:16.604945 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.587500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.15.18.126:33619)
0929 02:58:16.587722 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.587728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.589088 (+ 1360us) server_negotiation.cc:407] Connection header received
0929 02:58:16.590088 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.590092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.590149 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.590199 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.591593 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.592635 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.594533 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.595206 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.595323 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.599121 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.599125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.599129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.599160 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.601845 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.602758 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.602763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.602767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.602928 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.603366 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.603371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.603375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.603768 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.603922 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.604368 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.604573 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:58:16.604957 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.587992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:33619 (local address 127.0.0.1:45334)
0929 02:58:16.588958 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.589014 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:16.589922 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:16.590363 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:16.590368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:16.591087 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:16.591433 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.591442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.592811 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.592816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.594408 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.594421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.595357 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.595362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.595544 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.596115 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:16.596144 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:16.598918 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:16.601998 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.602011 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.602018 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.602647 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.603052 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.603056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.603060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.603235 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.603878 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:16.603885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:16.604074 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 02:58:16.604220 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 02:58:16.604556 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":575,"threads_started":1}
I20260929 02:58:16.606446 29064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:58:16.608096 28916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:16.609403 28916 log.cc:824] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:16.610082 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.583134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.15.18.124:46821)
0929 02:58:16.589479 (+ 6345us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.589485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.589516 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:58:16.589580 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.589584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.589650 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.589748 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.591658 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.592603 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.594575 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.595012 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.595113 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.600683 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.600690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.600695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.600762 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.606598 (+ 5836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.607658 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.607663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.607666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.607806 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.608272 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.608277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.608280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.608670 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.608946 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.609266 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.609487 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6249,"thread_start_us":6109,"threads_started":1}
I20260929 02:58:16.613307 28916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e: No bootstrap required, opened a new log
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650696, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:16.622758 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.607174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.15.18.125:41567)
0929 02:58:16.607336 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.607342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.607412 (+ 70us) server_negotiation.cc:407] Connection header received
0929 02:58:16.607582 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.607587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.607666 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.607757 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.608590 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.609564 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.611160 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.611800 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.611916 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.615567 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.615574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.615578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.615621 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.619008 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.620302 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.620309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.620312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.620476 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.620955 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.620960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.620963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.621307 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.621432 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.622016 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.622262 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:58:16.622741 28916 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:16.623304 28916 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:16.623479 28916 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b244072da044ffb68da8886fa4f43e, State: Initialized, Role: FOLLOWER
I20260929 02:58:16.623965 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.607082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:41567 (local address 127.0.0.1:41724)
0929 02:58:16.607333 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.607350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:16.607454 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:16.607919 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:16.607924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:16.608286 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:16.608441 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.608450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.609763 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.609767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.610986 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.610998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.612112 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.612118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.612304 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.612886 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:16.612907 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:16.615330 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:16.619622 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.619627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.619632 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.620167 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.620649 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.620654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.620658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.620829 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.621526 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:16.621531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:16.621744 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 02:58:16.623240 (+ 1496us) client_negotiation.cc:240] Negotiation successful
0929 02:58:16.623642 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260929 02:58:16.624344 28916 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:16.625094 29064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:58:16.628487 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.611765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.15.18.126:33619)
0929 02:58:16.611924 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.611929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.611949 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:58:16.612111 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.612117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.612178 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.612255 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.613562 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.614388 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.615725 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.616442 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.616580 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.622047 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.622053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.622058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.622094 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.625652 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.626485 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.626491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.626494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.626678 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.627069 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.627074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.627078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.627362 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.627708 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.627969 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.628175 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:58:16.631358 29071 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } } }
I20260929 02:58:16.631872 29071 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:58:16.633347 28916 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650696, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
I20260929 02:58:16.648346 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.625901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:43988)
0929 02:58:16.626356 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.626374 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:16.626463 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:16.634021 (+ 7558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:16.634025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:16.634503 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:16.634694 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.634707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.635976 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.635981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.637225 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.637236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.638071 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.638077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.638244 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.638744 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:16.638767 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:16.641243 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:16.645659 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.645665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.645669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.646176 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.646572 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.646577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.646581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.646710 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.647306 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:16.647311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:16.647453 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 02:58:16.647606 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 02:58:16.647940 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353}
I20260929 02:58:16.649005 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.630001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.15.18.125:41567)
0929 02:58:16.630162 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.630167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.630249 (+ 82us) server_negotiation.cc:407] Connection header received
0929 02:58:16.630410 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.630416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.630479 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.630563 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.631863 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.632900 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.634666 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.635373 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.635474 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.642478 (+ 7004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.642485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.642489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.642530 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.645806 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.646682 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.646687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.646691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.646848 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.647304 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.647308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.647311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.647667 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.647909 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.648470 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.648689 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:58:16.649341 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.626207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.15.18.124:46821)
0929 02:58:16.633629 (+ 7422us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.633635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.633653 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:58:16.633717 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.633722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.633790 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.633881 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.634846 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.635838 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.637346 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.637921 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.638027 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.641578 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.641585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.641590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.641624 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.645504 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.646316 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.646321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.646325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.646458 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.646805 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.646809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.646813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.647195 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.648718 (+ 1523us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.648840 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.649053 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7298}
W20260929 02:58:16.652575 29082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:58:16.652794 29082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:58:16.654206 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:58:16.654945 29064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d: Bootstrap starting.
I20260929 02:58:16.645582 28849 master_runner.cc:186] Error getting master registration for 127.15.18.125:41567: OK, instance_id { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" instance_seqno: 1790650696082616 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:58:16.657081 28849 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:58:16.659164 29064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:16.660458 29064 log.cc:824] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:16.674608 29064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d: No bootstrap required, opened a new log
I20260929 02:58:16.690755 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.660917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.15.18.124:46821)
0929 02:58:16.667851 (+ 6934us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.667857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.673224 (+ 5367us) server_negotiation.cc:407] Connection header received
0929 02:58:16.673707 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.673714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.673793 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.673863 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.676311 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.677430 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.679584 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.680273 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.680370 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.684173 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.684179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.684185 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.684236 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.687361 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.688223 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.688228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.688232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.688365 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.688852 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.688857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.688860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.689214 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.689482 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.690103 (+ 621us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.690329 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6796}
I20260929 02:58:16.690776 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.659664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.15.18.126:33619)
0929 02:58:16.659846 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.659852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.670820 (+ 10968us) server_negotiation.cc:407] Connection header received
0929 02:58:16.671965 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.671971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.672056 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.672176 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.674691 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.675871 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.678124 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.678794 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.678889 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.682736 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.682743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.682749 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.682790 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.686831 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.687994 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.687999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.688003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.688174 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.688906 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.688911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.688915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.689301 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.689537 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.690163 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.690402 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:58:16.693212 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.660604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.15.18.125:41567)
0929 02:58:16.660771 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.660776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.676596 (+ 15820us) server_negotiation.cc:407] Connection header received
0929 02:58:16.677084 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.677090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.677169 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.677235 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.679585 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.680592 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.682675 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.683299 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.683394 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.686581 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.686587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.686592 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.686636 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.689943 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.690755 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.690760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.690763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.690917 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.691361 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.691365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.691368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.691760 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.692113 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.692662 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.692878 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:58:16.697436 29064 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:16.697903 29064 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:16.698028 29064 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc164ccddd3495b87cb71d88158b54d, State: Initialized, Role: FOLLOWER
I20260929 02:58:16.698698 29064 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:16.699746 29092 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } } }
I20260929 02:58:16.700073 29092 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:58:16.701098 29064 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:58:16.714226 29103 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:58:16.714447 29103 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:58:16.712487 28997 master_runner.cc:186] Error getting master registration for 127.15.18.125:41567: OK, instance_id { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" instance_seqno: 1790650696082616 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:58:16.721773 28997 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:58:16.804174 28990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } attempt: 5
W20260929 02:58:16.805852 29018 tablet.cc:2369] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:58:16 dist-test-slave-q5f0 krb5kdc[28836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650696, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
I20260929 02:58:16.823421 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.806975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.15.18.124:46821)
0929 02:58:16.807148 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:16.807154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:16.807171 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:58:16.807224 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:16.807229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:16.807303 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:16.807399 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:16.808559 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.809764 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.811539 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.812317 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.812435 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.816414 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:16.816422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:16.816427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:16.816466 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:16.819934 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.820800 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.820807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.820811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.820958 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:16.821397 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:16.821403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:16.821406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:16.821800 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:16.821975 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:16.822805 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 02:58:16.823045 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:58:16.824448 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:16.804851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:46821 (local address 127.0.0.1:44012)
0929 02:58:16.806081 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:16.806101 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:16.806210 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:16.807604 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:16.807609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:16.808172 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:16.808388 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.808401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.809917 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.809923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.811364 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:16.811380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:16.812477 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:16.812484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:16.812727 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:16.813397 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:16.813420 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:16.816181 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:16.820127 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.820133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.820137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.820664 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.821081 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:16.821086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:16.821090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:16.821259 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:16.822047 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:16.822052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:16.822215 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 02:58:16.822567 (+ 352us) client_negotiation.cc:240] Negotiation successful
0929 02:58:16.824064 (+ 1497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077}
I20260929 02:58:16.830701 28990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee: Bootstrap starting.
I20260929 02:58:16.834695 28990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:16.836031 28990 log.cc:824] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:16.841430 28990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee: No bootstrap required, opened a new log
I20260929 02:58:16.852327 28990 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:16.852818 28990 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:16.852941 28990 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc97cf31d4a1475d85887f50f628e7ee, State: Initialized, Role: FOLLOWER
I20260929 02:58:16.853607 28990 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:16.854583 29106 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } } }
I20260929 02:58:16.855038 29106 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [sys.catalog]: This master's current role is: FOLLOWER
W20260929 02:58:16.855173 28870 tablet.cc:2369] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:58:16.855934 28990 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:58:16.873014 28923 master_runner.cc:263] No leader master found from master bc97cf31d4a1475d85887f50f628e7ee
I20260929 02:58:16.873189 28923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 02:58:16.877952 29117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:58:16.878103 29117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:58:16.998581 29092 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:58:16.998924 29092 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:17.001788 29092 leader_election.cc:288] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6b244072da044ffb68da8886fa4f43e (127.15.18.126:33619), bc97cf31d4a1475d85887f50f628e7ee (127.15.18.125:41567)
I20260929 02:58:17.003041 28965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecc164ccddd3495b87cb71d88158b54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc97cf31d4a1475d85887f50f628e7ee" is_pre_election: true
I20260929 02:58:17.003751 28965 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc164ccddd3495b87cb71d88158b54d in term 0.
I20260929 02:58:17.003866 29071 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:58:17.004158 29071 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:17.004441 29014 leader_election.cc:302] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc97cf31d4a1475d85887f50f628e7ee, ecc164ccddd3495b87cb71d88158b54d; no voters:
I20260929 02:58:17.004895 29092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:58:17.005028 29092 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:58:17.005179 29092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:17.005620 29071 leader_election.cc:288] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc97cf31d4a1475d85887f50f628e7ee (127.15.18.125:41567), ecc164ccddd3495b87cb71d88158b54d (127.15.18.124:46821)
I20260929 02:58:17.006531 28891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecc164ccddd3495b87cb71d88158b54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b244072da044ffb68da8886fa4f43e" is_pre_election: true
I20260929 02:58:17.006949 28891 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc164ccddd3495b87cb71d88158b54d in term 0.
I20260929 02:58:17.008138 29039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6b244072da044ffb68da8886fa4f43e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc164ccddd3495b87cb71d88158b54d" is_pre_election: true
I20260929 02:58:17.008818 28965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6b244072da044ffb68da8886fa4f43e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc97cf31d4a1475d85887f50f628e7ee" is_pre_election: true
I20260929 02:58:17.009044 28965 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b244072da044ffb68da8886fa4f43e in term 0.
I20260929 02:58:17.009562 28866 leader_election.cc:302] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc97cf31d4a1475d85887f50f628e7ee, c6b244072da044ffb68da8886fa4f43e; no voters:
I20260929 02:58:17.010049 29071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:58:17.010200 29071 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:58:17.010309 29071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:17.017168 29071 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:17.018507 28965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6b244072da044ffb68da8886fa4f43e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc97cf31d4a1475d85887f50f628e7ee"
I20260929 02:58:17.018759 28965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:17.022208 28965 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b244072da044ffb68da8886fa4f43e in term 1.
I20260929 02:58:17.022858 28866 leader_election.cc:302] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc97cf31d4a1475d85887f50f628e7ee, c6b244072da044ffb68da8886fa4f43e; no voters:
I20260929 02:58:17.026628 29038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6b244072da044ffb68da8886fa4f43e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc164ccddd3495b87cb71d88158b54d"
I20260929 02:58:17.025260 29092 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:17.028321 29071 leader_election.cc:288] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [CANDIDATE]: Term 1 election: Requested vote from peers bc97cf31d4a1475d85887f50f628e7ee (127.15.18.125:41567), ecc164ccddd3495b87cb71d88158b54d (127.15.18.124:46821)
I20260929 02:58:17.030782 29039 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6b244072da044ffb68da8886fa4f43e in current term 1: Already voted for candidate ecc164ccddd3495b87cb71d88158b54d in this term.
I20260929 02:58:17.032188 29071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:58:17.033267 28891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecc164ccddd3495b87cb71d88158b54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b244072da044ffb68da8886fa4f43e"
I20260929 02:58:17.034066 29071 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [term 1 LEADER]: Becoming Leader. State: Replica: c6b244072da044ffb68da8886fa4f43e, State: Running, Role: LEADER
I20260929 02:58:17.035049 28965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecc164ccddd3495b87cb71d88158b54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc97cf31d4a1475d85887f50f628e7ee"
I20260929 02:58:17.035473 28965 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ecc164ccddd3495b87cb71d88158b54d in current term 1: Already voted for candidate c6b244072da044ffb68da8886fa4f43e in this term.
I20260929 02:58:17.036783 29071 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } }
I20260929 02:58:17.037264 29092 leader_election.cc:288] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [CANDIDATE]: Term 1 election: Requested vote from peers c6b244072da044ffb68da8886fa4f43e (127.15.18.126:33619), bc97cf31d4a1475d85887f50f628e7ee (127.15.18.125:41567)
I20260929 02:58:17.048448 29015 leader_election.cc:302] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecc164ccddd3495b87cb71d88158b54d; no voters: bc97cf31d4a1475d85887f50f628e7ee, c6b244072da044ffb68da8886fa4f43e
I20260929 02:58:17.049068 29092 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 02:58:17.049876 29119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b244072da044ffb68da8886fa4f43e. Latest consensus state: current_term: 1 leader_uuid: "c6b244072da044ffb68da8886fa4f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } } }
I20260929 02:58:17.050397 29119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:17.055680 29121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:58:17.062798 29121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260929 02:58:17.073163 28944 tablet.cc:2369] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:58:17.089576 28965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [term 1 FOLLOWER]: Refusing update from remote peer c6b244072da044ffb68da8886fa4f43e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:58:17.091169 29071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:58:17.093216 29039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [term 1 FOLLOWER]: Refusing update from remote peer c6b244072da044ffb68da8886fa4f43e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:58:17.095088 29071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:58:17.145161 29106 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b244072da044ffb68da8886fa4f43e. Latest consensus state: current_term: 1 leader_uuid: "c6b244072da044ffb68da8886fa4f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } } }
I20260929 02:58:17.145509 29106 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:58:17.151646 29123 mvcc.cc:205] Tried to move back new op lower bound from 7334505255264337920 to 7334505255075676160. Current Snapshot: MvccSnapshot[applied={T|T < 7334505255264337920}]
I20260929 02:58:17.152877 29106 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6b244072da044ffb68da8886fa4f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } } }
I20260929 02:58:17.153128 29106 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:58:17.154533 29092 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b244072da044ffb68da8886fa4f43e. Latest consensus state: current_term: 1 leader_uuid: "c6b244072da044ffb68da8886fa4f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } } }
I20260929 02:58:17.154793 29092 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:58:17.160327 29124 mvcc.cc:205] Tried to move back new op lower bound from 7334505255264337920 to 7334505255075676160. Current Snapshot: MvccSnapshot[applied={T|T < 7334505255264337920}]
I20260929 02:58:17.165298 29119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6b244072da044ffb68da8886fa4f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } } }
I20260929 02:58:17.165592 29119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:17.166021 29119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6b244072da044ffb68da8886fa4f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } } }
I20260929 02:58:17.166246 29119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:17.166982 29121 catalog_manager.cc:1347] Generated new cluster ID: 8a07eed4ea5e4008b843c77ed1354e6f
I20260929 02:58:17.167073 29121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:58:17.170523 29092 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6b244072da044ffb68da8886fa4f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b244072da044ffb68da8886fa4f43e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33619 } } peers { permanent_uuid: "bc97cf31d4a1475d85887f50f628e7ee" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 41567 } } peers { permanent_uuid: "ecc164ccddd3495b87cb71d88158b54d" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 46821 } } }
I20260929 02:58:17.170825 29092 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:58:17.184952 29121 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:58:17.186776 29121 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:58:17.198197 29121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6b244072da044ffb68da8886fa4f43e: Generated new TSK 0
I20260929 02:58:17.199401 29121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:58:17.716439 29103 catalog_manager.cc:1259] Loaded cluster ID: 8a07eed4ea5e4008b843c77ed1354e6f
I20260929 02:58:17.716542 29103 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d: loading cluster ID for follower catalog manager: success
I20260929 02:58:17.718901 29103 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d: acquiring CA information for follower catalog manager: success
I20260929 02:58:17.720649 29103 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ecc164ccddd3495b87cb71d88158b54d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:58:17.880144 29117 catalog_manager.cc:1259] Loaded cluster ID: 8a07eed4ea5e4008b843c77ed1354e6f
I20260929 02:58:17.880251 29117 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee: loading cluster ID for follower catalog manager: success
I20260929 02:58:17.882665 29117 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee: acquiring CA information for follower catalog manager: success
I20260929 02:58:17.884449 29117 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc97cf31d4a1475d85887f50f628e7ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:58:18.286983 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 28849
I20260929 02:58:18.288257 28911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:18.401134 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 28849
I20260929 02:58:18.423350 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 28923
I20260929 02:58:18.424680 28985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:18.533384 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 28923
I20260929 02:58:18.549742 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 28997
I20260929 02:58:18.550942 29059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:18.656872 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 28997
2026-09-29T02:58:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4673 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:58:18 dist-test-slave-q5f0 krb5kdc[29139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:58:18 dist-test-slave-q5f0 krb5kdc[29139](info): set up 2 sockets
Sep 29 02:58:18 dist-test-slave-q5f0 krb5kdc[29139](info): commencing operation
krb5kdc: starting...
W20260929 02:58:19.676050 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.009s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:58:19 dist-test-slave-q5f0 krb5kdc[29139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:58:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:58:19.887652 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:33393
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:33393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:20.204986 29152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:20.205281 29152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:20.205396 29152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:20.212850 29152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:58:20.212975 29152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:20.213035 29152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:20.213117 29152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:58:20.213178 29152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:58:20.223117 29152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:33393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:33393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:20.224475 29152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:20.225817 29152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:20.237864 29158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:20.237908 29157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:20.237929 29160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:20.238473 29152 server_base.cc:1034] running on GCE node
I20260929 02:58:20.239446 29152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:20.240945 29152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:20.242143 29152 hybrid_clock.cc:648] HybridClock initialized: now 1790650700242086 us; error 45 us; skew 500 ppm
I20260929 02:58:20.242489 29152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:20 dist-test-slave-q5f0 krb5kdc[29139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650700, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:20.245841 29152 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:58:20.247381 29152 webserver.cc:460] Webserver started at http://127.15.18.126:43047/ using document root <none> and password file <none>
I20260929 02:58:20.248088 29152 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:20.248203 29152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:20.248485 29152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:20.251410 29152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "9b33748fbac44780b3235b26edb26676"
format_stamp: "Formatted at 2026-09-29 02:58:20 on dist-test-slave-q5f0"
server_key: "0f0906ecf7f73c3c2e3507e2c83cdd0e"
server_key_iv: "90fc5e6845fea22a025339736dab1d8f"
server_key_version: "encryptionkey@0"
I20260929 02:58:20.252166 29152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "9b33748fbac44780b3235b26edb26676"
format_stamp: "Formatted at 2026-09-29 02:58:20 on dist-test-slave-q5f0"
server_key: "0f0906ecf7f73c3c2e3507e2c83cdd0e"
server_key_iv: "90fc5e6845fea22a025339736dab1d8f"
server_key_version: "encryptionkey@0"
I20260929 02:58:20.257200 29152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:58:20.260645 29167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:20.261951 29152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:58:20.262117 29152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "9b33748fbac44780b3235b26edb26676"
format_stamp: "Formatted at 2026-09-29 02:58:20 on dist-test-slave-q5f0"
server_key: "0f0906ecf7f73c3c2e3507e2c83cdd0e"
server_key_iv: "90fc5e6845fea22a025339736dab1d8f"
server_key_version: "encryptionkey@0"
I20260929 02:58:20.262295 29152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:20.274405 29152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:20.278683 29152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:20.278942 29152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:20.296568 29152 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:33393
I20260929 02:58:20.296618 29218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:33393 every 8 connection(s)
I20260929 02:58:20.297950 29152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:58:20.301697 29219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:20.307927 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 29152
I20260929 02:58:20.308127 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:58:20.308735 15433 external_mini_cluster.cc:1428] Setting key 25232cc6dddd1616041f2dc8e216f724
I20260929 02:58:20.312214 29219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676: Bootstrap starting.
I20260929 02:58:20.315135 29219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:20.316419 29219 log.cc:824] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:20.319336 29219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676: No bootstrap required, opened a new log
Sep 29 02:58:20 dist-test-slave-q5f0 krb5kdc[29139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:20.328402 29219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b33748fbac44780b3235b26edb26676" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33393 } }
I20260929 02:58:20.328796 29219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:20.328907 29219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b33748fbac44780b3235b26edb26676, State: Initialized, Role: FOLLOWER
I20260929 02:58:20.329638 29219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b33748fbac44780b3235b26edb26676" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33393 } }
I20260929 02:58:20.329879 29219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:58:20.330015 29219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:58:20.330180 29219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:20.332322 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:20.310578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.15.18.126:33393)
0929 02:58:20.311313 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:20.311325 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:20.311364 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:58:20.313125 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:20.313147 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:20.313511 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:20.313868 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:20.315449 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:20.317322 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:20.318784 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:20.319847 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:20.320024 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:20.324190 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:20.324205 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:20.324228 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:20.324276 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:20.328136 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:20.328921 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:20.328930 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:20.328939 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:20.329131 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:20.329551 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:20.329555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:20.329557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:20.330156 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:20.330406 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:20.330818 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 02:58:20.331189 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":335,"threads_started":1}
I20260929 02:58:20.332971 29219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b33748fbac44780b3235b26edb26676" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33393 } }
I20260929 02:58:20.333457 29219 leader_election.cc:302] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b33748fbac44780b3235b26edb26676; no voters:
I20260929 02:58:20.334081 29219 leader_election.cc:288] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:58:20.334163 29224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:58:20.335764 29224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [term 1 LEADER]: Becoming Leader. State: Replica: 9b33748fbac44780b3235b26edb26676, State: Running, Role: LEADER
I20260929 02:58:20.336122 29219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:58:20.336328 29224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b33748fbac44780b3235b26edb26676" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33393 } }
I20260929 02:58:20.336585 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:58:20.340705 29225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b33748fbac44780b3235b26edb26676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b33748fbac44780b3235b26edb26676" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33393 } } }
I20260929 02:58:20.341143 29225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:20.341487 29226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b33748fbac44780b3235b26edb26676. Latest consensus state: current_term: 1 leader_uuid: "9b33748fbac44780b3235b26edb26676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b33748fbac44780b3235b26edb26676" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33393 } } }
I20260929 02:58:20.341821 29226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676 [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:20.344465 29232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:58:20.348724 29232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:58:20.357107 29232 catalog_manager.cc:1347] Generated new cluster ID: 90ff8401727f4d42881ca4de8c251a51
I20260929 02:58:20.357215 29232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:58:20.388780 29232 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:58:20.390305 29232 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:58:20.402693 29232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b33748fbac44780b3235b26edb26676: Generated new TSK 0
I20260929 02:58:20.403551 29232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:58:20.449942 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 29152
I20260929 02:58:20.451248 29214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:20.549664 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 29152
2026-09-29T02:58:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1890 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:58:20 dist-test-slave-q5f0 krb5kdc[29247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:58:20 dist-test-slave-q5f0 krb5kdc[29247](info): set up 2 sockets
Sep 29 02:58:20 dist-test-slave-q5f0 krb5kdc[29247](info): commencing operation
krb5kdc: starting...
W20260929 02:58:21.552160 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:58:21 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:58:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:58:21.767381 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46199
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:46199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:22.076406 29260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:22.076717 29260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:22.076857 29260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:22.084100 29260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:58:22.084214 29260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:22.084282 29260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:22.084357 29260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:58:22.084419 29260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:58:22.094277 29260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:46199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:22.095574 29260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:22.096870 29260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:22.107995 29265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:22.108034 29268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:22.108069 29266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:22.109187 29260 server_base.cc:1034] running on GCE node
I20260929 02:58:22.109979 29260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:22.111377 29260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:22.112576 29260 hybrid_clock.cc:648] HybridClock initialized: now 1790650702112528 us; error 36 us; skew 500 ppm
I20260929 02:58:22.112907 29260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:22 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650702, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:22.123371 29260 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:58:22.124984 29260 webserver.cc:460] Webserver started at http://127.15.18.126:38799/ using document root <none> and password file <none>
I20260929 02:58:22.125628 29260 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:22.125736 29260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:22.125983 29260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:22.129014 29260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2"
format_stamp: "Formatted at 2026-09-29 02:58:22 on dist-test-slave-q5f0"
server_key: "ea7bca598c70ed56c2826ea4b3a8ca03"
server_key_iv: "3c42297396471ca50d0456b194569cb1"
server_key_version: "encryptionkey@0"
I20260929 02:58:22.129773 29260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2"
format_stamp: "Formatted at 2026-09-29 02:58:22 on dist-test-slave-q5f0"
server_key: "ea7bca598c70ed56c2826ea4b3a8ca03"
server_key_iv: "3c42297396471ca50d0456b194569cb1"
server_key_version: "encryptionkey@0"
I20260929 02:58:22.135012 29260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:58:22.138716 29275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:22.140105 29260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:58:22.140257 29260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2"
format_stamp: "Formatted at 2026-09-29 02:58:22 on dist-test-slave-q5f0"
server_key: "ea7bca598c70ed56c2826ea4b3a8ca03"
server_key_iv: "3c42297396471ca50d0456b194569cb1"
server_key_version: "encryptionkey@0"
I20260929 02:58:22.140441 29260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:22.161530 29260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:22.165755 29260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:22.166016 29260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:22.183617 29260 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:46199
I20260929 02:58:22.183684 29326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:46199 every 8 connection(s)
I20260929 02:58:22.184963 29260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:58:22.187502 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 29260
I20260929 02:58:22.187752 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:58:22.188592 15433 external_mini_cluster.cc:1428] Setting key c051e073a65ac77ce8a8448e9982e029
I20260929 02:58:22.189123 29327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:22.203042 29327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Bootstrap starting.
Sep 29 02:58:22 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:22.206614 29327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:22.207705 29327 log.cc:824] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:22.210976 29327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: No bootstrap required, opened a new log
I20260929 02:58:22.211890 29330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:22.190540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.15.18.126:46199)
0929 02:58:22.191271 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:22.191283 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:22.191323 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:58:22.193600 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:22.193622 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:22.193999 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:22.194329 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:22.196068 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:22.197354 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:22.199114 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:22.199870 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:22.200003 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:22.204737 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:22.204749 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:22.204764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:22.204799 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:22.207694 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:22.208511 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:22.208520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:22.208528 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:22.208739 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:22.209194 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:22.209197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:22.209200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:22.209714 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:22.209938 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:22.210425 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 02:58:22.210755 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":320,"threads_started":1}
I20260929 02:58:22.218405 29327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:22.218756 29327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:22.218853 29327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbd75b7ca1d49d1923a60a4a32a86d2, State: Initialized, Role: FOLLOWER
I20260929 02:58:22.219388 29327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:22.219646 29327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:58:22.219770 29327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:58:22.219924 29327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:22.222060 29327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:22.222551 29327 leader_election.cc:302] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dbd75b7ca1d49d1923a60a4a32a86d2; no voters:
I20260929 02:58:22.223120 29327 leader_election.cc:288] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:58:22.223263 29332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:58:22.223869 29332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 1 LEADER]: Becoming Leader. State: Replica: 3dbd75b7ca1d49d1923a60a4a32a86d2, State: Running, Role: LEADER
I20260929 02:58:22.224363 29332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:22.225106 29327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:58:22.226832 29333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } } }
I20260929 02:58:22.226890 29334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dbd75b7ca1d49d1923a60a4a32a86d2. Latest consensus state: current_term: 1 leader_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } } }
I20260929 02:58:22.227187 29333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:22.227327 29334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:22.229379 29337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:58:22.234862 29337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:58:22.244899 29337 catalog_manager.cc:1347] Generated new cluster ID: df23d9fa9027401fa1044656283d4818
I20260929 02:58:22.245028 29337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:58:22.276115 29337 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:58:22.277868 29337 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:58:22.292375 29337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Generated new TSK 0
I20260929 02:58:22.293237 29337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:58:22.357494 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46199
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:22.672853 29353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:22.673151 29353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:22.673286 29353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:22.680583 29353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:22.680707 29353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:22.680858 29353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:58:22.690637 29353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:22.691962 29353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:22.693228 29353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:22.705505 29361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:22.705646 29358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:22.705607 29359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:22.706722 29353 server_base.cc:1034] running on GCE node
I20260929 02:58:22.707326 29353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:22.708307 29353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:22.709502 29353 hybrid_clock.cc:648] HybridClock initialized: now 1790650702709457 us; error 42 us; skew 500 ppm
I20260929 02:58:22.709803 29353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:22 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650702, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:22.712909 29353 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:58:22.714282 29353 webserver.cc:460] Webserver started at http://127.15.18.65:38253/ using document root <none> and password file <none>
I20260929 02:58:22.714917 29353 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:22.715013 29353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:22.715260 29353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:22.718078 29353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "bc522fe93c2142dbb5b620d031653bd8"
format_stamp: "Formatted at 2026-09-29 02:58:22 on dist-test-slave-q5f0"
server_key: "cb9efd712faff4865d997ca4a504247f"
server_key_iv: "8483325a5d39ec22724f82e1d2ea829b"
server_key_version: "encryptionkey@0"
I20260929 02:58:22.718812 29353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "bc522fe93c2142dbb5b620d031653bd8"
format_stamp: "Formatted at 2026-09-29 02:58:22 on dist-test-slave-q5f0"
server_key: "cb9efd712faff4865d997ca4a504247f"
server_key_iv: "8483325a5d39ec22724f82e1d2ea829b"
server_key_version: "encryptionkey@0"
I20260929 02:58:22.723869 29353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:58:22.727542 29368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:22.728891 29353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:58:22.729035 29353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "bc522fe93c2142dbb5b620d031653bd8"
format_stamp: "Formatted at 2026-09-29 02:58:22 on dist-test-slave-q5f0"
server_key: "cb9efd712faff4865d997ca4a504247f"
server_key_iv: "8483325a5d39ec22724f82e1d2ea829b"
server_key_version: "encryptionkey@0"
I20260929 02:58:22.729223 29353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:22.746501 29353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:22.750597 29353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:22.750856 29353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:22.751847 29353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:22.753155 29353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:22.753224 29353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:22.753331 29353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:22.753382 29353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:22.782910 29353 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:41655
I20260929 02:58:22.782943 29480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:41655 every 8 connection(s)
I20260929 02:58:22.784426 29353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:58:22.790531 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 29353
I20260929 02:58:22.790735 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:58:22.791400 15433 external_mini_cluster.cc:1428] Setting key e1b4d75b0585deac77b3568e8f2e0e55
Sep 29 02:58:22 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650702, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:22.804163 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:22.786830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:43289 (local address 127.15.18.126:46199)
0929 02:58:22.787264 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:22.787268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:22.788111 (+ 843us) server_negotiation.cc:407] Connection header received
0929 02:58:22.789848 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:22.789853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:22.789936 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:22.790031 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:22.791569 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:22.792489 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:22.794370 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:22.794986 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:22.795113 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:22.798470 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:22.798476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:22.798480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:22.798516 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:22.801218 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:22.802053 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:22.802057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:22.802059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:22.802182 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:22.802593 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:22.802597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:22.802599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:22.802843 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:22.802948 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:22.803759 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 02:58:22.803924 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":222,"threads_started":1}
I20260929 02:58:22.805465 29484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:22.787284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:43289)
0929 02:58:22.787964 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:22.788001 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:22.789588 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:22.790284 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:22.790293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:22.790833 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:22.791376 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:22.791393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:22.792701 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:22.792706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:22.794219 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:22.794231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:22.795186 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:22.795191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:22.795388 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:22.796032 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:22.796057 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:22.798246 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:22.801354 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:22.801364 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:22.801383 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:22.801933 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:22.802276 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:22.802281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:22.802284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:22.802500 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:22.802947 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:22.802955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:22.803359 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 02:58:22.803682 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 02:58:22.804190 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":266,"threads_started":1}
I20260929 02:58:22.806969 29481 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46199
I20260929 02:58:22.807250 29481 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:22.807999 29481 heartbeater.cc:511] Master 127.15.18.126:46199 requested a full tablet report, sending...
I20260929 02:58:22.809926 29292 ts_manager.cc:194] Registered new tserver with Master: bc522fe93c2142dbb5b620d031653bd8 (127.15.18.65:41655)
I20260929 02:58:22.811301 29292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:43289
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:58:22.842316 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46199
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:23.157747 29487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:23.158023 29487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:23.158114 29487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:23.165318 29487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:23.165431 29487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:23.165544 29487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:58:23.175297 29487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:23.176594 29487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:23.177822 29487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:23.191289 29493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:23.191735 29492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:23.191866 29495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:23.192229 29487 server_base.cc:1034] running on GCE node
I20260929 02:58:23.192816 29487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:23.193785 29487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:23.195034 29487 hybrid_clock.cc:648] HybridClock initialized: now 1790650703194948 us; error 51 us; skew 500 ppm
I20260929 02:58:23.195358 29487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:23 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650703, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:23.198534 29487 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:58:23.199987 29487 webserver.cc:460] Webserver started at http://127.15.18.66:46361/ using document root <none> and password file <none>
I20260929 02:58:23.200640 29487 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:23.200754 29487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:23.201022 29487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:23.203884 29487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "36d543d2933b448499d3117db7ef3e8d"
format_stamp: "Formatted at 2026-09-29 02:58:23 on dist-test-slave-q5f0"
server_key: "65f91f5e7b69dbadcf8a71d0f9a97e0d"
server_key_iv: "7bc91859f0c9633e0b0bfd613403f927"
server_key_version: "encryptionkey@0"
I20260929 02:58:23.204617 29487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "36d543d2933b448499d3117db7ef3e8d"
format_stamp: "Formatted at 2026-09-29 02:58:23 on dist-test-slave-q5f0"
server_key: "65f91f5e7b69dbadcf8a71d0f9a97e0d"
server_key_iv: "7bc91859f0c9633e0b0bfd613403f927"
server_key_version: "encryptionkey@0"
I20260929 02:58:23.209493 29487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:58:23.212847 29502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:23.214118 29487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:58:23.214277 29487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "36d543d2933b448499d3117db7ef3e8d"
format_stamp: "Formatted at 2026-09-29 02:58:23 on dist-test-slave-q5f0"
server_key: "65f91f5e7b69dbadcf8a71d0f9a97e0d"
server_key_iv: "7bc91859f0c9633e0b0bfd613403f927"
server_key_version: "encryptionkey@0"
I20260929 02:58:23.214453 29487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:23.230786 29487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:23.234822 29487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:23.235080 29487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:23.236060 29487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:23.237380 29487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:23.237455 29487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:23.237571 29487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:23.237625 29487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:23.266758 29487 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:44763
I20260929 02:58:23.266793 29614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:44763 every 8 connection(s)
I20260929 02:58:23.268006 29487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:58:23.272948 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 29487
I20260929 02:58:23.273121 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:58:23.273541 15433 external_mini_cluster.cc:1428] Setting key 4fd335745143f187e5a05bfad3835427
Sep 29 02:58:23 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650703, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:23.287493 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.270511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:35481 (local address 127.15.18.126:46199)
0929 02:58:23.270715 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:23.270721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:23.271885 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 02:58:23.273707 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:23.273714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:23.273799 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:23.273912 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:23.275740 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.276456 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.277936 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.278337 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.278442 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.281645 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:23.281650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:23.281653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:23.281684 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:23.284560 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:23.285335 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:23.285339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:23.285342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:23.285471 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:23.285929 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:23.285932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:23.285934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:23.286205 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:23.286329 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:23.287079 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 02:58:23.287252 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:58:23.288776 29617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.271001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:35481)
0929 02:58:23.271739 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:23.271779 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:23.273435 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:23.274096 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:23.274107 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:23.274741 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:23.275499 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.275548 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.276605 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.276610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.277799 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.277811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.278538 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.278544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.278767 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.279365 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:23.279386 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:23.281434 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:23.284680 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:23.284690 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:23.284715 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:23.285204 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:23.285562 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:23.285567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:23.285571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:23.285804 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:23.286315 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:23.286324 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:23.286722 (+ 398us) client_negotiation.cc:769] Sending connection context
0929 02:58:23.286985 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 02:58:23.287486 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":272,"threads_started":1}
I20260929 02:58:23.290189 29615 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46199
I20260929 02:58:23.290457 29615 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:23.291158 29615 heartbeater.cc:511] Master 127.15.18.126:46199 requested a full tablet report, sending...
I20260929 02:58:23.292675 29292 ts_manager.cc:194] Registered new tserver with Master: 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763)
I20260929 02:58:23.293552 29292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:35481
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:58:23.323164 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46199
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:23.634016 29620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:23.634303 29620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:23.634440 29620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:23.641732 29620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:23.641866 29620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:23.642015 29620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:58:23.651868 29620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:23.653110 29620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:23.654415 29620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:23.667047 29625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:23.667011 29628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:23.667135 29626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:23.667960 29620 server_base.cc:1034] running on GCE node
I20260929 02:58:23.668527 29620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:23.669484 29620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:23.670713 29620 hybrid_clock.cc:648] HybridClock initialized: now 1790650703670646 us; error 54 us; skew 500 ppm
I20260929 02:58:23.671141 29620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:23 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650703, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:23.674327 29620 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:58:23.675920 29620 webserver.cc:460] Webserver started at http://127.15.18.67:42669/ using document root <none> and password file <none>
I20260929 02:58:23.676595 29620 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:23.676702 29620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:23.676961 29620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:23.680008 29620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "58b578abe8e54b2c984de01880f27307"
format_stamp: "Formatted at 2026-09-29 02:58:23 on dist-test-slave-q5f0"
server_key: "9d8367d4d00ec37abcf22ddc6ac52242"
server_key_iv: "1aeedf18244bdce454db02f74cc96f59"
server_key_version: "encryptionkey@0"
I20260929 02:58:23.680792 29620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "58b578abe8e54b2c984de01880f27307"
format_stamp: "Formatted at 2026-09-29 02:58:23 on dist-test-slave-q5f0"
server_key: "9d8367d4d00ec37abcf22ddc6ac52242"
server_key_iv: "1aeedf18244bdce454db02f74cc96f59"
server_key_version: "encryptionkey@0"
I20260929 02:58:23.686267 29620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:58:23.689772 29635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:23.691102 29620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:58:23.691233 29620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "58b578abe8e54b2c984de01880f27307"
format_stamp: "Formatted at 2026-09-29 02:58:23 on dist-test-slave-q5f0"
server_key: "9d8367d4d00ec37abcf22ddc6ac52242"
server_key_iv: "1aeedf18244bdce454db02f74cc96f59"
server_key_version: "encryptionkey@0"
I20260929 02:58:23.691396 29620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:23.708221 29620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:23.712301 29620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:23.712531 29620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:23.713375 29620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:23.714663 29620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:23.714742 29620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:23.714835 29620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:23.714876 29620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:23.744742 29620 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:46743
I20260929 02:58:23.744825 29747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:46743 every 8 connection(s)
I20260929 02:58:23.745932 29620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:58:23.753652 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 29620
I20260929 02:58:23.753878 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:58:23.754421 15433 external_mini_cluster.cc:1428] Setting key b7a94dfefa24e95096d807f640ef0868
Sep 29 02:58:23 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650703, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:23.764483 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.748245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:46199 (local address 127.15.18.126:46199)
0929 02:58:23.748456 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:23.748463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:23.749621 (+ 1158us) server_negotiation.cc:407] Connection header received
0929 02:58:23.751331 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:23.751336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:23.751405 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:23.751490 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:23.752852 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.753584 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.755727 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.756228 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.756330 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.759613 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:23.759618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:23.759621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:23.759652 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:23.762013 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:23.762633 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:23.762636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:23.762638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:23.762743 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:23.763062 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:23.763065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:23.763067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:23.763315 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:23.763419 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:23.764059 (+ 640us) server_negotiation.cc:299] Negotiation successful
0929 02:58:23.764244 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:58:23.765383 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.748790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:46199)
0929 02:58:23.749461 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:23.749504 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:23.751088 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:23.751689 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:23.751697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:23.752183 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:23.752720 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.752737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.753755 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.753760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.755569 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.755583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.756442 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.756450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.756645 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.757339 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:23.757372 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:23.759413 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:23.762134 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:23.762141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:23.762153 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:23.762550 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:23.762821 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:23.762824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:23.762826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:23.762989 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:23.763401 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:23.763406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:23.763758 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 02:58:23.763972 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 02:58:23.764340 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":251,"threads_started":1}
I20260929 02:58:23.766601 29748 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46199
I20260929 02:58:23.766903 29748 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:23.767652 29748 heartbeater.cc:511] Master 127.15.18.126:46199 requested a full tablet report, sending...
I20260929 02:58:23.769121 29292 ts_manager.cc:194] Registered new tserver with Master: 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743)
I20260929 02:58:23.769680 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:58:23.770167 29292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:46199
I20260929 02:58:23.786208 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.774078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.15.18.126:46199)
0929 02:58:23.774256 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:23.774262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:23.774607 (+ 345us) server_negotiation.cc:407] Connection header received
0929 02:58:23.774733 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:23.774737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:23.774796 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:23.774873 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:23.776182 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.776945 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.778294 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.778681 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.778769 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.781015 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:23.781036 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:23.781039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:23.781071 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:23.783592 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:23.784330 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:23.784335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:23.784337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:23.784471 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:23.784864 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:23.784868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:23.784870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:23.785219 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:23.785406 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:23.785783 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 02:58:23.785949 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:58:23.790230 29292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:58:23.796928 29292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:58:23.814258 29481 heartbeater.cc:503] Master 127.15.18.126:46199 was elected leader, sending a full tablet report...
I20260929 02:58:23.818648 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.807919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46743 (local address 127.0.0.1:55250)
0929 02:58:23.808814 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:23.808857 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:23.809012 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:23.810097 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:23.810104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:23.810136 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:23.810542 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.810556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.812907 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.812912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.815459 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.815481 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.817488 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.817495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.817742 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.817804 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 02:58:23.817920 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 02:58:23.818133 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":4,"thread_start_us":559,"threads_started":1}
I20260929 02:58:23.818945 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.808131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:44763 (local address 127.0.0.1:56712)
0929 02:58:23.809840 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:23.809857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:23.809960 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:23.811339 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:23.811344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:23.811371 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:23.811838 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.811850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.814029 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.814034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.816037 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.816053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.818078 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.818085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.818315 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.818353 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:58:23.818418 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 02:58:23.818567 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1598,"thread_start_us":1476,"threads_started":1}
I20260929 02:58:23.820149 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.808706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.15.18.66:44763)
0929 02:58:23.810725 (+ 2019us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:23.810734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:23.810762 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:58:23.810861 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:23.810869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:23.811120 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:23.812038 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:23.812297 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.813868 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.816279 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.817811 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.819274 (+ 1463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.819526 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:23.819606 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 02:58:23.819813 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1874,"thread_start_us":1775,"threads_started":1}
I20260929 02:58:23.820367 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.808166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.15.18.67:46743)
0929 02:58:23.809566 (+ 1400us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:23.809573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:23.809596 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:58:23.809670 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:23.809675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:23.809866 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:23.810732 (+ 866us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:23.811010 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.812739 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.815721 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.817270 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.819470 (+ 2200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.819770 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:23.819854 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 02:58:23.820033 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1263,"thread_start_us":1143,"threads_started":1}
I20260929 02:58:23.823841 29683 tablet_service.cc:1467] Processing CreateTablet for tablet 534f21132c1b43dbba71836233e5ba5e (DEFAULT_TABLE table=test-table [id=3d6e317bff8242878d16c5e30f445248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:23.824050 29550 tablet_service.cc:1467] Processing CreateTablet for tablet 534f21132c1b43dbba71836233e5ba5e (DEFAULT_TABLE table=test-table [id=3d6e317bff8242878d16c5e30f445248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:23.825215 29550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534f21132c1b43dbba71836233e5ba5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:23.825237 29683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534f21132c1b43dbba71836233e5ba5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:23.828497 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.809971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41655 (local address 127.0.0.1:47796)
0929 02:58:23.818901 (+ 8930us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:23.818919 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:23.819019 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:23.821804 (+ 2785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:23.821809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:23.821834 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:23.821991 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.822002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.824105 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.824111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.825941 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.825957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.827709 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.827716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.827899 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.827934 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:58:23.828044 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 02:58:23.828194 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8840,"thread_start_us":10608,"threads_started":1}
I20260929 02:58:23.828557 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.810334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.15.18.65:41655)
0929 02:58:23.821194 (+ 10860us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:23.821202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:23.821230 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:58:23.821325 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:23.821333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:23.821590 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:23.822179 (+ 589us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:23.822453 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.823951 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.826141 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.827546 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.827680 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.827936 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:23.828078 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 02:58:23.828239 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10678,"thread_start_us":10525,"threads_started":1}
I20260929 02:58:23.832568 29416 tablet_service.cc:1467] Processing CreateTablet for tablet 534f21132c1b43dbba71836233e5ba5e (DEFAULT_TABLE table=test-table [id=3d6e317bff8242878d16c5e30f445248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:23.834031 29416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534f21132c1b43dbba71836233e5ba5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:23.837149 29766 tablet_bootstrap.cc:492] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: Bootstrap starting.
I20260929 02:58:23.839480 29767 tablet_bootstrap.cc:492] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: Bootstrap starting.
I20260929 02:58:23.841648 29766 tablet_bootstrap.cc:654] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:23.842289 29767 tablet_bootstrap.cc:654] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:23.842942 29766 log.cc:824] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:23.843195 29767 log.cc:824] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:23.845692 29766 tablet_bootstrap.cc:492] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: No bootstrap required, opened a new log
I20260929 02:58:23.845937 29766 ts_tablet_manager.cc:1397] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:58:23.846032 29767 tablet_bootstrap.cc:492] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: No bootstrap required, opened a new log
I20260929 02:58:23.846290 29767 ts_tablet_manager.cc:1397] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260929 02:58:23.849085 29770 tablet_bootstrap.cc:492] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: Bootstrap starting.
I20260929 02:58:23.852475 29770 tablet_bootstrap.cc:654] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:23.853400 29766 raft_consensus.cc:348] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:23.853696 29770 log.cc:824] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:23.853777 29766 raft_consensus.cc:374] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:23.853863 29766 raft_consensus.cc:729] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d543d2933b448499d3117db7ef3e8d, State: Initialized, Role: FOLLOWER
I20260929 02:58:23.854471 29766 consensus_queue.cc:260] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:23.856081 29615 heartbeater.cc:503] Master 127.15.18.126:46199 was elected leader, sending a full tablet report...
I20260929 02:58:23.855897 29767 raft_consensus.cc:348] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:23.856339 29767 raft_consensus.cc:374] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:23.856441 29767 raft_consensus.cc:729] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b578abe8e54b2c984de01880f27307, State: Initialized, Role: FOLLOWER
I20260929 02:58:23.856984 29770 tablet_bootstrap.cc:492] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: No bootstrap required, opened a new log
I20260929 02:58:23.857021 29767 consensus_queue.cc:260] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:23.857260 29770 ts_tablet_manager.cc:1397] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:58:23.858022 29766 ts_tablet_manager.cc:1428] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260929 02:58:23.858112 29748 heartbeater.cc:503] Master 127.15.18.126:46199 was elected leader, sending a full tablet report...
I20260929 02:58:23.858675 29767 ts_tablet_manager.cc:1428] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260929 02:58:23.865764 29770 raft_consensus.cc:348] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:23.866108 29770 raft_consensus.cc:374] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:23.866205 29770 raft_consensus.cc:729] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc522fe93c2142dbb5b620d031653bd8, State: Initialized, Role: FOLLOWER
I20260929 02:58:23.866703 29770 consensus_queue.cc:260] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:23.868275 29770 ts_tablet_manager.cc:1428] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 02:58:23.872731 29773 raft_consensus.cc:482] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:58:23.872962 29773 raft_consensus.cc:504] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:23.874428 29773 leader_election.cc:288] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763), bc522fe93c2142dbb5b620d031653bd8 (127.15.18.65:41655)
I20260929 02:58:23.880692 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.874735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:48039 (local address 127.15.18.66:44763)
0929 02:58:23.874899 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:23.874903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:23.874920 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:58:23.875079 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:23.875084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:23.875158 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:23.875250 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:23.875799 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.876788 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.878677 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.879823 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.879937 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.879988 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:23.880337 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 02:58:23.880438 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:58:23.880746 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.874615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:44763 (local address 127.15.18.67:48039)
0929 02:58:23.874783 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:23.874802 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:23.874915 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:23.875393 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:23.875398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:23.875430 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:23.875641 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.875649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.876918 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.876923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.878511 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.878523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.879960 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.879966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.880153 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.880189 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:58:23.880297 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 02:58:23.880425 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":40}
I20260929 02:58:23.881385 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.875119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:51461 (local address 127.15.18.65:41655)
0929 02:58:23.875300 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:23.875306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:23.875705 (+ 399us) server_negotiation.cc:407] Connection header received
0929 02:58:23.875834 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:23.875838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:23.875888 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:23.875948 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:23.876510 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.877416 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.879436 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.880411 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.880546 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.880608 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:23.881014 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 02:58:23.881158 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:58:23.881474 29775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.874963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41655 (local address 127.15.18.67:51461)
0929 02:58:23.875615 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:23.875630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:23.875736 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:23.875979 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:23.875983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:23.876008 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:23.876376 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.876388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.877637 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.877640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.879278 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:23.879293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.880651 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.880657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:23.880824 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.880855 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:58:23.880983 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 02:58:23.881104 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":1,"thread_start_us":302,"threads_started":1}
I20260929 02:58:23.881673 29570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534f21132c1b43dbba71836233e5ba5e" candidate_uuid: "58b578abe8e54b2c984de01880f27307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d543d2933b448499d3117db7ef3e8d" is_pre_election: true
I20260929 02:58:23.882194 29570 raft_consensus.cc:2454] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58b578abe8e54b2c984de01880f27307 in term 0.
I20260929 02:58:23.882524 29436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534f21132c1b43dbba71836233e5ba5e" candidate_uuid: "58b578abe8e54b2c984de01880f27307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc522fe93c2142dbb5b620d031653bd8" is_pre_election: true
I20260929 02:58:23.882820 29639 leader_election.cc:302] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d543d2933b448499d3117db7ef3e8d, 58b578abe8e54b2c984de01880f27307; no voters:
I20260929 02:58:23.883064 29436 raft_consensus.cc:2454] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58b578abe8e54b2c984de01880f27307 in term 0.
I20260929 02:58:23.883200 29773 raft_consensus.cc:2781] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:58:23.883342 29773 raft_consensus.cc:482] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:58:23.883428 29773 raft_consensus.cc:3037] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:23.885565 29773 raft_consensus.cc:504] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:23.886119 29773 leader_election.cc:288] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [CANDIDATE]: Term 1 election: Requested vote from peers 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763), bc522fe93c2142dbb5b620d031653bd8 (127.15.18.65:41655)
I20260929 02:58:23.886616 29570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534f21132c1b43dbba71836233e5ba5e" candidate_uuid: "58b578abe8e54b2c984de01880f27307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d543d2933b448499d3117db7ef3e8d"
I20260929 02:58:23.886636 29436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534f21132c1b43dbba71836233e5ba5e" candidate_uuid: "58b578abe8e54b2c984de01880f27307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc522fe93c2142dbb5b620d031653bd8"
I20260929 02:58:23.886798 29570 raft_consensus.cc:3037] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:23.886791 29436 raft_consensus.cc:3037] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:23.889230 29570 raft_consensus.cc:2454] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58b578abe8e54b2c984de01880f27307 in term 1.
I20260929 02:58:23.889372 29436 raft_consensus.cc:2454] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58b578abe8e54b2c984de01880f27307 in term 1.
I20260929 02:58:23.889658 29639 leader_election.cc:302] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d543d2933b448499d3117db7ef3e8d, 58b578abe8e54b2c984de01880f27307; no voters:
I20260929 02:58:23.889948 29773 raft_consensus.cc:2781] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:58:23.890367 29773 raft_consensus.cc:686] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 1 LEADER]: Becoming Leader. State: Replica: 58b578abe8e54b2c984de01880f27307, State: Running, Role: LEADER
I20260929 02:58:23.890817 29773 consensus_queue.cc:237] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:23.895831 29292 catalog_manager.cc:5495] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58b578abe8e54b2c984de01880f27307 (127.15.18.67). New cstate: current_term: 1 leader_uuid: "58b578abe8e54b2c984de01880f27307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } health_report { overall_health: UNKNOWN } } }
I20260929 02:58:23.920312 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.915284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.15.18.67:46743)
0929 02:58:23.915456 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:23.915460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:23.915476 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:58:23.915589 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:23.915593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:23.915668 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:23.915787 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:58:23.916091 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.916823 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.918305 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.918689 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.918798 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.919326 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:58:23.919846 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:58:23.919953 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:23.920011 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 02:58:23.920073 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:58:23.928515 29570 raft_consensus.cc:1261] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d [term 1 FOLLOWER]: Refusing update from remote peer 58b578abe8e54b2c984de01880f27307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:58:23.928515 29436 raft_consensus.cc:1261] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Refusing update from remote peer 58b578abe8e54b2c984de01880f27307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:58:23.929646 29776 consensus_queue.cc:1035] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:58:23.929978 29773 consensus_queue.cc:1035] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:58:23.940214 29779 mvcc.cc:205] Tried to move back new op lower bound from 7334505283280572416 to 7334505283140706304. Current Snapshot: MvccSnapshot[applied={T|T < 7334505283280572416}]
I20260929 02:58:23.956346 29292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:58:23.956584 29292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:58:23.960170 29292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Sending DeleteTablet for 3 replicas of tablet 534f21132c1b43dbba71836233e5ba5e
I20260929 02:58:23.961256 29683 tablet_service.cc:1514] Processing DeleteTablet for tablet 534f21132c1b43dbba71836233e5ba5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:23 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:55250
I20260929 02:58:23.961465 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 29353
I20260929 02:58:23.961663 29550 tablet_service.cc:1514] Processing DeleteTablet for tablet 534f21132c1b43dbba71836233e5ba5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:23 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:56712
I20260929 02:58:23.962634 29416 tablet_service.cc:1514] Processing DeleteTablet for tablet 534f21132c1b43dbba71836233e5ba5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:23 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:47796
I20260929 02:58:23.972703 29789 tablet_replica.cc:330] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: stopping tablet replica
I20260929 02:58:23.973172 29790 tablet_replica.cc:330] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: stopping tablet replica
I20260929 02:58:23.973212 29789 raft_consensus.cc:2229] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:58:23.973694 29790 raft_consensus.cc:2229] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:58:23.973891 29789 raft_consensus.cc:2258] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:23.974292 29790 raft_consensus.cc:2258] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:23.976172 29789 ts_tablet_manager.cc:1905] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:23.976445 29790 ts_tablet_manager.cc:1905] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:23.979429 29792 tablet_replica.cc:330] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: stopping tablet replica
I20260929 02:58:23.979868 29792 raft_consensus.cc:2229] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:58:23.980337 29792 raft_consensus.cc:2258] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:23.983322 29790 ts_tablet_manager.cc:1918] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:23.983363 29789 ts_tablet_manager.cc:1918] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:23.983577 29789 log.cc:1196] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/534f21132c1b43dbba71836233e5ba5e
Sep 29 02:58:23 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:58:23.983592 29790 log.cc:1196] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/534f21132c1b43dbba71836233e5ba5e
I20260929 02:58:23.984180 29789 ts_tablet_manager.cc:1939] T 534f21132c1b43dbba71836233e5ba5e P 58b578abe8e54b2c984de01880f27307: Deleting consensus metadata
I20260929 02:58:23.984321 29790 ts_tablet_manager.cc:1939] T 534f21132c1b43dbba71836233e5ba5e P 36d543d2933b448499d3117db7ef3e8d: Deleting consensus metadata
I20260929 02:58:23.985594 29792 ts_tablet_manager.cc:1905] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:23.985931 29277 catalog_manager.cc:4841] TS 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743): tablet 534f21132c1b43dbba71836233e5ba5e (table test-table [id=3d6e317bff8242878d16c5e30f445248]) successfully deleted
I20260929 02:58:23.986457 29279 catalog_manager.cc:4841] TS 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763): tablet 534f21132c1b43dbba71836233e5ba5e (table test-table [id=3d6e317bff8242878d16c5e30f445248]) successfully deleted
I20260929 02:58:23.991261 29792 ts_tablet_manager.cc:1918] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:23.991410 29792 log.cc:1196] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/534f21132c1b43dbba71836233e5ba5e
I20260929 02:58:23.991854 29792 ts_tablet_manager.cc:1939] T 534f21132c1b43dbba71836233e5ba5e P bc522fe93c2142dbb5b620d031653bd8: Deleting consensus metadata
I20260929 02:58:23.993492 29279 catalog_manager.cc:4841] TS bc522fe93c2142dbb5b620d031653bd8 (127.15.18.65:41655): tablet 534f21132c1b43dbba71836233e5ba5e (table test-table [id=3d6e317bff8242878d16c5e30f445248]) successfully deleted
I20260929 02:58:23.995074 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:23.962243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.15.18.65:41655)
0929 02:58:23.962406 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:23.962412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:23.971896 (+ 9484us) server_negotiation.cc:407] Connection header received
0929 02:58:23.972691 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:23.972697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:23.972778 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:23.972887 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:23.974822 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.975939 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.977812 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:23.978438 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:23.978530 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:23.987827 (+ 9297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:23.987846 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:23.987865 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:23.987913 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:23.992001 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:23.992860 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:23.992869 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:23.992873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:23.993028 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:23.993475 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:23.993480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:23.993483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:23.993938 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:23.994272 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:23.994498 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 02:58:23.994718 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:58:23.995893 29476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:24.085759 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 29353
I20260929 02:58:24.100571 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 29487
Sep 29 02:58:24 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:58:24.117162 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:24.101506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.15.18.66:44763)
0929 02:58:24.101717 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:24.101724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:24.101748 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:58:24.101867 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:24.101874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:24.101965 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:24.102094 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:24.103328 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:24.104489 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:24.105965 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:24.106631 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:24.106777 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:24.111494 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:24.111542 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:24.111563 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:24.111606 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:24.114300 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:24.115253 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:24.115260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:24.115262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:24.115383 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:24.115815 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:24.115819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:24.115821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:24.116173 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:24.116404 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:24.116743 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 02:58:24.116914 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:58:24.117928 29610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:24.224722 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 29487
I20260929 02:58:24.238785 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 29620
Sep 29 02:58:24 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:58:24.256774 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:24.239812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.15.18.67:46743)
0929 02:58:24.240044 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:24.240051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:24.240077 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:58:24.240155 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:24.240162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:24.240265 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:24.240407 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:24.241681 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:24.242882 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:24.244448 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:24.245212 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:24.245380 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:24.250002 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:24.250017 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:24.250033 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:24.250083 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:24.253462 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:24.254406 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:24.254413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:24.254416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:24.254582 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:24.255058 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:24.255065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:24.255067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:24.255435 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:24.255719 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:24.256145 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 02:58:24.256332 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 02:58:24.257577 29743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:24.378479 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 29620
I20260929 02:58:24.393352 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 29260
I20260929 02:58:24.394423 29322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:24.502645 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 29260
I20260929 02:58:24.519400 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46199
--webserver_interface=127.15.18.126
--webserver_port=38799
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:46199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:24.831148 29797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:24.831421 29797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:24.831527 29797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:24.838644 29797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:58:24.838752 29797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:24.838806 29797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:24.838860 29797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:58:24.838902 29797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:58:24.848686 29797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:46199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=38799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:24.849943 29797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:24.851131 29797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:24.872341 29803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:24.872300 29804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:24.872265 29806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:24.873148 29797 server_base.cc:1034] running on GCE node
I20260929 02:58:24.873957 29797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:24.875478 29797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:24.876698 29797 hybrid_clock.cc:648] HybridClock initialized: now 1790650704876642 us; error 49 us; skew 500 ppm
I20260929 02:58:24.877079 29797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:24 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650704, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:24.880440 29797 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:58:24.882052 29797 webserver.cc:460] Webserver started at http://127.15.18.126:38799/ using document root <none> and password file <none>
I20260929 02:58:24.882771 29797 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:24.882902 29797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:24.888652 29797 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:58:24.891628 29813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:24.892947 29797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:58:24.893100 29797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2"
format_stamp: "Formatted at 2026-09-29 02:58:22 on dist-test-slave-q5f0"
server_key: "ea7bca598c70ed56c2826ea4b3a8ca03"
server_key_iv: "3c42297396471ca50d0456b194569cb1"
server_key_version: "encryptionkey@0"
I20260929 02:58:24.894104 29797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:24.909356 29797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:24.913539 29797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:24.913801 29797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:24.931192 29797 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:46199
I20260929 02:58:24.931214 29864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:46199 every 8 connection(s)
I20260929 02:58:24.932700 29797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:58:24.939985 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 29797
I20260929 02:58:24.940248 29865 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:58:24.941123 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:41655
--local_ip_for_outbound_sockets=127.15.18.65
--tserver_master_addrs=127.15.18.126:46199
--webserver_port=38253
--webserver_interface=127.15.18.65
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:58:24.945150 29865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Bootstrap starting.
I20260929 02:58:24.966333 29865 log.cc:824] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:24.979589 29865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:58:24.980145 29865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Bootstrap complete.
I20260929 02:58:24.989553 29865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:24.990422 29865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbd75b7ca1d49d1923a60a4a32a86d2, State: Initialized, Role: FOLLOWER
I20260929 02:58:24.991063 29865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:24.991308 29865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:58:24.991443 29865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:58:24.991639 29865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:58:24.995635 29865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:24.996177 29865 leader_election.cc:302] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dbd75b7ca1d49d1923a60a4a32a86d2; no voters:
I20260929 02:58:24.996790 29865 leader_election.cc:288] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:58:24.996944 29869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:58:24.998155 29869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 2 LEADER]: Becoming Leader. State: Replica: 3dbd75b7ca1d49d1923a60a4a32a86d2, State: Running, Role: LEADER
I20260929 02:58:24.998849 29869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:24.999203 29865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:58:25.001451 29871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dbd75b7ca1d49d1923a60a4a32a86d2. Latest consensus state: current_term: 2 leader_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } } }
I20260929 02:58:25.001529 29870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } } }
I20260929 02:58:25.001922 29870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:25.001820 29871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:25.016601 29876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:58:25.024799 29876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:58:25.026144 29876 catalog_manager.cc:1259] Loaded cluster ID: df23d9fa9027401fa1044656283d4818
I20260929 02:58:25.026255 29876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:58:25.029282 29876 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 02:58:25.046077 29876 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 02:58:25.055178 29876 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:58:25.056838 29876 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Loaded TSK: 0
I20260929 02:58:25.058305 29876 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:58:25.277933 29867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:25.278203 29867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:25.278335 29867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:25.285724 29867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:25.285832 29867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:25.285996 29867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:58:25.295921 29867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:41655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=38253
--tserver_master_addrs=127.15.18.126:46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:25.297274 29867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:25.298648 29867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:25.313776 29895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:25.313894 29892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:25.313938 29893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:25.314764 29867 server_base.cc:1034] running on GCE node
I20260929 02:58:25.315629 29867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:25.316543 29867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:25.317745 29867 hybrid_clock.cc:648] HybridClock initialized: now 1790650705317674 us; error 62 us; skew 500 ppm
I20260929 02:58:25.318027 29867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:25 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650705, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:25.320996 29867 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:58:25.322440 29867 webserver.cc:460] Webserver started at http://127.15.18.65:38253/ using document root <none> and password file <none>
I20260929 02:58:25.323099 29867 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:25.323221 29867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:25.328464 29867 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:58:25.331146 29902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:25.332593 29867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:58:25.332741 29867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "bc522fe93c2142dbb5b620d031653bd8"
format_stamp: "Formatted at 2026-09-29 02:58:22 on dist-test-slave-q5f0"
server_key: "cb9efd712faff4865d997ca4a504247f"
server_key_iv: "8483325a5d39ec22724f82e1d2ea829b"
server_key_version: "encryptionkey@0"
I20260929 02:58:25.333681 29867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:25.347036 29867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:25.351042 29867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:25.351290 29867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:25.352325 29867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:25.353590 29867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:25.353688 29867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:25.353789 29867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:25.353855 29867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:25.387733 29867 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:41655
I20260929 02:58:25.387980 30014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:41655 every 8 connection(s)
I20260929 02:58:25.389184 29867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:58:25.395053 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 29867
I20260929 02:58:25.395687 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:44763
--local_ip_for_outbound_sockets=127.15.18.66
--tserver_master_addrs=127.15.18.126:46199
--webserver_port=46361
--webserver_interface=127.15.18.66
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:58:25 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650705, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:25.411545 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:25.392066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:60127)
0929 02:58:25.392789 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:25.392831 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:25.394424 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:25.395812 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:25.395823 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:25.396497 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:25.397285 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:25.397309 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:25.398998 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:25.399002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:25.400251 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:25.400264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:25.401264 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:25.401272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:25.401507 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:25.402102 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:25.402124 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:25.404219 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:25.407373 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:25.407381 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:25.407396 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:25.407870 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:25.408285 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:25.408290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:25.408294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:25.408471 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:25.409163 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:25.409171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:25.409504 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 02:58:25.409729 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 02:58:25.410261 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":288,"threads_started":1}
I20260929 02:58:25.411553 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:25.391849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:60127 (local address 127.15.18.126:46199)
0929 02:58:25.392405 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:25.392422 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:25.392953 (+ 531us) server_negotiation.cc:407] Connection header received
0929 02:58:25.394951 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:25.394963 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:25.395298 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:25.395614 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:25.397487 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:25.398834 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:25.400385 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:25.401038 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:25.401204 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:25.404475 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:25.404482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:25.404496 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:25.404530 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:25.407241 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:25.408004 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:25.408011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:25.408016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:25.408171 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:25.408584 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:25.408588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:25.408590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:25.409036 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:25.409207 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:25.409953 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 02:58:25.410337 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":214,"threads_started":1}
I20260929 02:58:25.414157 30015 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46199
I20260929 02:58:25.414430 30015 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:25.415104 30015 heartbeater.cc:511] Master 127.15.18.126:46199 requested a full tablet report, sending...
I20260929 02:58:25.417150 29830 ts_manager.cc:194] Registered new tserver with Master: bc522fe93c2142dbb5b620d031653bd8 (127.15.18.65:41655)
I20260929 02:58:25.418598 29830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:60127
W20260929 02:58:25.707563 30019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:25.707849 30019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:25.707985 30019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:25.715198 30019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:25.715317 30019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:25.715468 30019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:58:25.725242 30019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:44763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=46361
--tserver_master_addrs=127.15.18.126:46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:25.726488 30019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:25.727775 30019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:25.740163 30024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:25.740203 30027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:25.740127 30025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:25.740867 30019 server_base.cc:1034] running on GCE node
I20260929 02:58:25.741475 30019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:25.742389 30019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:25.743625 30019 hybrid_clock.cc:648] HybridClock initialized: now 1790650705743567 us; error 47 us; skew 500 ppm
I20260929 02:58:25.743932 30019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:25 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650705, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:25.746930 30019 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:58:25.748354 30019 webserver.cc:460] Webserver started at http://127.15.18.66:46361/ using document root <none> and password file <none>
I20260929 02:58:25.749001 30019 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:25.749146 30019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:25.754205 30019 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 02:58:25.756821 30034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:25.758068 30019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:58:25.758220 30019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "36d543d2933b448499d3117db7ef3e8d"
format_stamp: "Formatted at 2026-09-29 02:58:23 on dist-test-slave-q5f0"
server_key: "65f91f5e7b69dbadcf8a71d0f9a97e0d"
server_key_iv: "7bc91859f0c9633e0b0bfd613403f927"
server_key_version: "encryptionkey@0"
I20260929 02:58:25.759168 30019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:25.786466 30019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:25.790560 30019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:25.790807 30019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:25.791812 30019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:25.793093 30019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:25.793197 30019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:25.793299 30019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:25.793373 30019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:25.822698 30019 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:44763
I20260929 02:58:25.822752 30146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:44763 every 8 connection(s)
I20260929 02:58:25.824112 30019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:58:25.826983 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 30019
I20260929 02:58:25.827735 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:46743
--local_ip_for_outbound_sockets=127.15.18.67
--tserver_master_addrs=127.15.18.126:46199
--webserver_port=42669
--webserver_interface=127.15.18.67
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:58:25 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650705, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:25.843554 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:25.826472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:33915 (local address 127.15.18.126:46199)
0929 02:58:25.826661 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:25.826666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:25.827967 (+ 1301us) server_negotiation.cc:407] Connection header received
0929 02:58:25.829836 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:25.829843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:25.829922 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:25.830022 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:25.831538 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:25.832472 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:25.833943 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:25.834368 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:25.834476 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:25.838228 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:25.838232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:25.838236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:25.838268 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:25.840634 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:25.841300 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:25.841305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:25.841307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:25.841425 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:25.841812 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:25.841817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:25.841820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:25.842081 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:25.842218 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:25.842949 (+ 731us) server_negotiation.cc:299] Negotiation successful
0929 02:58:25.843186 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:58:25.844533 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:25.827044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:33915)
0929 02:58:25.827799 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:25.827848 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:25.829560 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:25.830217 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:25.830226 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:25.830808 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:25.831357 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:25.831374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:25.832603 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:25.832607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:25.833798 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:25.833809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:25.834574 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:25.834581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:25.834827 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:25.835464 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:25.835490 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:25.838015 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:25.840764 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:25.840773 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:25.840791 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:25.841189 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:25.841526 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:25.841531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:25.841535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:25.841713 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:25.842212 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:25.842219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:25.842605 (+ 386us) client_negotiation.cc:769] Sending connection context
0929 02:58:25.842842 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 02:58:25.843358 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":286,"threads_started":1}
I20260929 02:58:25.845958 30147 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46199
I20260929 02:58:25.846240 30147 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:25.846907 30147 heartbeater.cc:511] Master 127.15.18.126:46199 requested a full tablet report, sending...
I20260929 02:58:25.848285 29830 ts_manager.cc:194] Registered new tserver with Master: 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763)
I20260929 02:58:25.849454 29830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:33915
W20260929 02:58:26.142371 30150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:26.142642 30150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:26.142735 30150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:26.150050 30150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:26.150173 30150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:26.150286 30150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:58:26.159948 30150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:46743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=42669
--tserver_master_addrs=127.15.18.126:46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:26.161167 30150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:26.162559 30150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:26.175163 30158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:26.175267 30156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:26.175209 30155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:26.175863 30150 server_base.cc:1034] running on GCE node
I20260929 02:58:26.176402 30150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:26.177260 30150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:26.178439 30150 hybrid_clock.cc:648] HybridClock initialized: now 1790650706178400 us; error 33 us; skew 500 ppm
I20260929 02:58:26.178728 30150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:26 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650706, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:26.181699 30150 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:58:26.183058 30150 webserver.cc:460] Webserver started at http://127.15.18.67:42669/ using document root <none> and password file <none>
I20260929 02:58:26.183770 30150 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:26.183888 30150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:26.188971 30150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:58:26.191525 30165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:26.192821 30150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:58:26.192976 30150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "58b578abe8e54b2c984de01880f27307"
format_stamp: "Formatted at 2026-09-29 02:58:23 on dist-test-slave-q5f0"
server_key: "9d8367d4d00ec37abcf22ddc6ac52242"
server_key_iv: "1aeedf18244bdce454db02f74cc96f59"
server_key_version: "encryptionkey@0"
I20260929 02:58:26.193930 30150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:26.218657 30150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:26.222945 30150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:26.223215 30150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:26.224234 30150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:26.225548 30150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:26.225637 30150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:26.225739 30150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:26.225801 30150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:26.255151 30150 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:46743
I20260929 02:58:26.255201 30277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:46743 every 8 connection(s)
I20260929 02:58:26.256532 30150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:58:26.259749 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 30150
Sep 29 02:58:26 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650706, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:26.275094 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.260823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.15.18.126:46199)
0929 02:58:26.261343 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.261349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.261813 (+ 464us) server_negotiation.cc:407] Connection header received
0929 02:58:26.261959 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.261965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.262042 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.262159 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:26.263984 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.264825 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.266162 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.266746 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.266839 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.269526 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:26.269531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:26.269534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:26.269564 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:26.272437 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:26.273208 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:26.273212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:26.273221 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:26.273347 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:26.273750 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:26.273753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:26.273756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:26.274103 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:26.274363 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.274575 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.274738 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":242,"threads_started":1}
I20260929 02:58:26.276867 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.258911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:58931 (local address 127.15.18.126:46199)
0929 02:58:26.259080 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.259086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.260322 (+ 1236us) server_negotiation.cc:407] Connection header received
0929 02:58:26.262448 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.262454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.262528 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.262635 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:26.264188 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.264956 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.266528 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.267061 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.267179 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.270786 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:26.270791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:26.270797 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:26.270838 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:26.273755 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:26.274585 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:26.274590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:26.274593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:26.274725 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:26.275174 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:26.275178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:26.275181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:26.275499 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:26.275648 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.276356 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.276549 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:58:26.277789 30280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.259336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:58931)
0929 02:58:26.260112 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.260165 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:26.262065 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:26.262832 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:26.262840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:26.263326 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:26.264009 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.264035 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.265102 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.265106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.266393 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.266406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.267266 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.267271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.267539 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.268101 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:26.268122 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:26.270570 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:26.273877 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:26.273886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:26.273902 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:26.274459 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:26.274833 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:26.274838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:26.274842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:26.275065 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:26.275666 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:26.275673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:26.276013 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 02:58:26.276236 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 02:58:26.276759 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":330,"threads_started":1}
I20260929 02:58:26.278931 30278 heartbeater.cc:347] Connected to a master server at 127.15.18.126:46199
I20260929 02:58:26.279173 30278 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:26.279836 30278 heartbeater.cc:511] Master 127.15.18.126:46199 requested a full tablet report, sending...
I20260929 02:58:26.281062 29830 ts_manager.cc:194] Registered new tserver with Master: 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743)
I20260929 02:58:26.281836 29830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:58931
I20260929 02:58:26.289548 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:58:26.296365 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.290561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.15.18.126:46199)
0929 02:58:26.290775 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.290781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.291189 (+ 408us) server_negotiation.cc:407] Connection header received
0929 02:58:26.291398 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.291401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.291573 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.291702 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:58:26.292096 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.292837 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.294274 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.294667 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.294775 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.295287 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:58:26.295811 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:58:26.295961 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.296049 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.296136 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:58:26.297240 29830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:58:26.304426 29830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:58:26.336396 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.315976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.15.18.66:44763)
0929 02:58:26.328255 (+ 12279us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.328261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.328288 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:58:26.328355 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.328360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.328583 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.328779 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:26.329517 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.331398 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.333564 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.334844 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.334962 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.335187 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.335850 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.336019 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12136,"thread_start_us":12001,"threads_started":1}
I20260929 02:58:26.336475 30290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.315363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46743 (local address 127.0.0.1:55272)
0929 02:58:26.327054 (+ 11691us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.327104 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:26.327318 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:26.328194 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:26.328200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:26.328228 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:26.328654 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.328669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.331108 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.331112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.333462 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.333477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.335213 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.335219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.335745 (+ 526us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.335777 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:58:26.335843 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 02:58:26.336026 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11444,"mutex_wait_us":18,"spinlock_wait_cycles":11264,"thread_start_us":12110,"threads_started":1}
I20260929 02:58:26.336802 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.315638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.15.18.67:46743)
0929 02:58:26.326536 (+ 10898us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.326545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.327551 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 02:58:26.327672 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.327680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.327954 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.328882 (+ 928us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:26.329184 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.330911 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.333662 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.334993 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.335983 (+ 990us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.336198 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.336264 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.336416 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10710,"thread_start_us":10543,"threads_started":1}
I20260929 02:58:26.337272 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.315780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:44763 (local address 127.0.0.1:56742)
0929 02:58:26.327065 (+ 11285us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.327106 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:26.327317 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:26.328878 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:26.328883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:26.328907 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:26.329299 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.329312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.331582 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.331586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.333424 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.333438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.335384 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.335391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.335607 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.335664 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 02:58:26.335789 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 02:58:26.336831 (+ 1042us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11138,"mutex_wait_us":28,"spinlock_wait_cycles":8064,"thread_start_us":10864,"threads_started":1}
I20260929 02:58:26.339007 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.329153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41655 (local address 127.0.0.1:47814)
0929 02:58:26.331280 (+ 2127us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.331298 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:26.331417 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:26.332092 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:26.332096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:26.332126 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:26.332590 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.332611 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.334407 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.334412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.336467 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.336481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.338270 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.338277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.338440 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.338479 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:58:26.338566 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 02:58:26.338681 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1980,"mutex_wait_us":19,"thread_start_us":1798,"threads_started":1}
I20260929 02:58:26.339437 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.327836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.15.18.65:41655)
0929 02:58:26.330929 (+ 3093us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.330937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.331562 (+ 625us) server_negotiation.cc:407] Connection header received
0929 02:58:26.331660 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.331667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.331890 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.332089 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:26.332802 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.334267 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.336631 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.337933 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.338047 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.338243 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.338950 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.339144 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2939,"thread_start_us":2811,"threads_started":1}
I20260929 02:58:26.341028 30082 tablet_service.cc:1467] Processing CreateTablet for tablet db4505e0c65c44f28a778b2c3d92e6eb (DEFAULT_TABLE table=test-table [id=d959a74b5fe2417782b8c84073a3eac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:26.341028 30213 tablet_service.cc:1467] Processing CreateTablet for tablet db4505e0c65c44f28a778b2c3d92e6eb (DEFAULT_TABLE table=test-table [id=d959a74b5fe2417782b8c84073a3eac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:26.342492 30213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4505e0c65c44f28a778b2c3d92e6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:26.342492 30082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4505e0c65c44f28a778b2c3d92e6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:26.342610 29950 tablet_service.cc:1467] Processing CreateTablet for tablet db4505e0c65c44f28a778b2c3d92e6eb (DEFAULT_TABLE table=test-table [id=d959a74b5fe2417782b8c84073a3eac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:26.343745 29950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4505e0c65c44f28a778b2c3d92e6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:26.355201 30294 tablet_bootstrap.cc:492] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Bootstrap starting.
I20260929 02:58:26.357362 30295 tablet_bootstrap.cc:492] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: Bootstrap starting.
I20260929 02:58:26.359803 30294 tablet_bootstrap.cc:654] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:26.360653 30295 tablet_bootstrap.cc:654] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:26.361111 30294 log.cc:824] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:26.361852 30295 log.cc:824] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:26.363942 30296 tablet_bootstrap.cc:492] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: Bootstrap starting.
I20260929 02:58:26.364555 30294 tablet_bootstrap.cc:492] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: No bootstrap required, opened a new log
I20260929 02:58:26.364826 30294 ts_tablet_manager.cc:1397] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 02:58:26.365568 30295 tablet_bootstrap.cc:492] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: No bootstrap required, opened a new log
I20260929 02:58:26.365792 30295 ts_tablet_manager.cc:1397] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260929 02:58:26.367295 30296 tablet_bootstrap.cc:654] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:26.368528 30296 log.cc:824] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:26.371797 30296 tablet_bootstrap.cc:492] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: No bootstrap required, opened a new log
I20260929 02:58:26.372048 30296 ts_tablet_manager.cc:1397] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260929 02:58:26.373195 30295 raft_consensus.cc:348] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:26.373509 30295 raft_consensus.cc:374] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:26.373621 30295 raft_consensus.cc:729] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d543d2933b448499d3117db7ef3e8d, State: Initialized, Role: FOLLOWER
I20260929 02:58:26.374157 30295 consensus_queue.cc:260] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:26.375198 30147 heartbeater.cc:503] Master 127.15.18.126:46199 was elected leader, sending a full tablet report...
I20260929 02:58:26.375072 30294 raft_consensus.cc:348] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:26.375459 30294 raft_consensus.cc:374] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:26.375562 30294 raft_consensus.cc:729] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc522fe93c2142dbb5b620d031653bd8, State: Initialized, Role: FOLLOWER
I20260929 02:58:26.375874 30295 ts_tablet_manager.cc:1428] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: Time spent starting tablet: real 0.010s user 0.009s sys 0.003s
I20260929 02:58:26.376199 30294 consensus_queue.cc:260] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:26.377790 30294 ts_tablet_manager.cc:1428] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260929 02:58:26.378248 30015 heartbeater.cc:503] Master 127.15.18.126:46199 was elected leader, sending a full tablet report...
I20260929 02:58:26.382474 30296 raft_consensus.cc:348] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:26.382846 30296 raft_consensus.cc:374] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:26.382967 30296 raft_consensus.cc:729] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b578abe8e54b2c984de01880f27307, State: Initialized, Role: FOLLOWER
I20260929 02:58:26.383543 30296 consensus_queue.cc:260] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:26.384650 30278 heartbeater.cc:503] Master 127.15.18.126:46199 was elected leader, sending a full tablet report...
I20260929 02:58:26.385172 30296 ts_tablet_manager.cc:1428] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
W20260929 02:58:26.391142 30016 tablet.cc:2369] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:58:26.395815 30301 raft_consensus.cc:482] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:58:26.396059 30301 raft_consensus.cc:504] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:26.397773 30301 leader_election.cc:288] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743), 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763)
I20260929 02:58:26.404074 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.398223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:49249 (local address 127.15.18.66:44763)
0929 02:58:26.398407 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.398413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.399075 (+ 662us) server_negotiation.cc:407] Connection header received
0929 02:58:26.399257 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.399263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.399326 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.399397 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:26.400011 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.400742 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.402449 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.403203 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.403340 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.403413 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.403724 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.403847 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:58:26.404175 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.398186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:35941 (local address 127.15.18.67:46743)
0929 02:58:26.398371 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.398377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.399206 (+ 829us) server_negotiation.cc:407] Connection header received
0929 02:58:26.399365 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.399370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.399427 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.399497 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:26.400185 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.400940 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.402528 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.403286 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.403421 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.403490 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.403809 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.403927 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:58:26.404196 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.398006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:44763 (local address 127.15.18.65:49249)
0929 02:58:26.398953 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.398968 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:26.399122 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:26.399424 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:26.399428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:26.399452 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:26.399876 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.399888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.400869 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.400873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.402303 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.402315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.403390 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.403396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.403583 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.403607 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:58:26.403699 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 02:58:26.403820 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":19,"thread_start_us":610,"threads_started":1}
I20260929 02:58:26.404379 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.398366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46743 (local address 127.15.18.65:35941)
0929 02:58:26.399109 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.399123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:26.399248 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:26.399555 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:26.399559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:26.399598 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:26.400025 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.400034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.401052 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.401058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.402394 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:26.402405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.403493 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.403501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:26.403674 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.403696 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:58:26.403772 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 02:58:26.403890 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":35,"thread_start_us":451,"threads_started":1}
I20260929 02:58:26.405110 30102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4505e0c65c44f28a778b2c3d92e6eb" candidate_uuid: "bc522fe93c2142dbb5b620d031653bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d543d2933b448499d3117db7ef3e8d" is_pre_election: true
I20260929 02:58:26.405280 30233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4505e0c65c44f28a778b2c3d92e6eb" candidate_uuid: "bc522fe93c2142dbb5b620d031653bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b578abe8e54b2c984de01880f27307" is_pre_election: true
I20260929 02:58:26.405699 30102 raft_consensus.cc:2454] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc522fe93c2142dbb5b620d031653bd8 in term 0.
I20260929 02:58:26.405718 30233 raft_consensus.cc:2454] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc522fe93c2142dbb5b620d031653bd8 in term 0.
I20260929 02:58:26.406338 29906 leader_election.cc:302] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d543d2933b448499d3117db7ef3e8d, bc522fe93c2142dbb5b620d031653bd8; no voters:
I20260929 02:58:26.406741 30301 raft_consensus.cc:2781] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:58:26.406898 30301 raft_consensus.cc:482] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:58:26.406993 30301 raft_consensus.cc:3037] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:26.409145 30301 raft_consensus.cc:504] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:26.409695 30301 leader_election.cc:288] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 1 election: Requested vote from peers 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743), 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763)
I20260929 02:58:26.410060 30233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4505e0c65c44f28a778b2c3d92e6eb" candidate_uuid: "bc522fe93c2142dbb5b620d031653bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58b578abe8e54b2c984de01880f27307"
I20260929 02:58:26.410108 30102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4505e0c65c44f28a778b2c3d92e6eb" candidate_uuid: "bc522fe93c2142dbb5b620d031653bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d543d2933b448499d3117db7ef3e8d"
I20260929 02:58:26.410233 30233 raft_consensus.cc:3037] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:26.410310 30102 raft_consensus.cc:3037] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:26.412494 30233 raft_consensus.cc:2454] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc522fe93c2142dbb5b620d031653bd8 in term 1.
I20260929 02:58:26.412515 30102 raft_consensus.cc:2454] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc522fe93c2142dbb5b620d031653bd8 in term 1.
I20260929 02:58:26.412846 29904 leader_election.cc:302] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58b578abe8e54b2c984de01880f27307, bc522fe93c2142dbb5b620d031653bd8; no voters:
I20260929 02:58:26.413117 30301 raft_consensus.cc:2781] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:58:26.413487 30301 raft_consensus.cc:686] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 LEADER]: Becoming Leader. State: Replica: bc522fe93c2142dbb5b620d031653bd8, State: Running, Role: LEADER
I20260929 02:58:26.413969 30301 consensus_queue.cc:237] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:26.418154 29829 catalog_manager.cc:5495] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc522fe93c2142dbb5b620d031653bd8 (127.15.18.65). New cstate: current_term: 1 leader_uuid: "bc522fe93c2142dbb5b620d031653bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } health_report { overall_health: HEALTHY } } }
I20260929 02:58:26.438386 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.433176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.15.18.65:41655)
0929 02:58:26.433341 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.433346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.433362 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:58:26.433473 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.433477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.433554 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.433669 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:58:26.434017 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.434691 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.436310 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.436763 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.436881 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.437331 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:58:26.437855 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:58:26.437966 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.438111 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.438179 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:58:26.446421 30102 raft_consensus.cc:1261] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d [term 1 FOLLOWER]: Refusing update from remote peer bc522fe93c2142dbb5b620d031653bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:58:26.446455 30233 raft_consensus.cc:1261] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307 [term 1 FOLLOWER]: Refusing update from remote peer bc522fe93c2142dbb5b620d031653bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:58:26.447314 30305 consensus_queue.cc:1035] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:58:26.447721 30301 consensus_queue.cc:1035] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:58:26.459940 30309 mvcc.cc:205] Tried to move back new op lower bound from 7334505293595017216 to 7334505293475012608. Current Snapshot: MvccSnapshot[applied={T|T < 7334505293595017216}]
I20260929 02:58:26.460429 30310 mvcc.cc:205] Tried to move back new op lower bound from 7334505293595017216 to 7334505293475012608. Current Snapshot: MvccSnapshot[applied={T|T < 7334505293595017216}]
I20260929 02:58:26.471009 29829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:58:26.471276 29829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:58:26.474745 29829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Sending DeleteTablet for 3 replicas of tablet db4505e0c65c44f28a778b2c3d92e6eb
I20260929 02:58:26.475804 30213 tablet_service.cc:1514] Processing DeleteTablet for tablet db4505e0c65c44f28a778b2c3d92e6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:26 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:55272
I20260929 02:58:26.475951 29950 tablet_service.cc:1514] Processing DeleteTablet for tablet db4505e0c65c44f28a778b2c3d92e6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:26 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:47814
I20260929 02:58:26.475946 30082 tablet_service.cc:1514] Processing DeleteTablet for tablet db4505e0c65c44f28a778b2c3d92e6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:26 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:56742
I20260929 02:58:26.476982 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 29867
I20260929 02:58:26.494563 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.477760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.15.18.65:41655)
0929 02:58:26.477922 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.477927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.477986 (+ 59us) server_negotiation.cc:407] Connection header received
0929 02:58:26.478159 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.478163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.478255 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.478357 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:26.480197 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.481491 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.483320 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.484012 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.484146 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.487557 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:26.487573 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:26.487592 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:26.487640 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:26.491561 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:26.492379 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:26.492389 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:26.492393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:26.492564 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:26.492931 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:26.492936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:26.492940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:26.493393 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:26.493716 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.493936 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.494159 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:58:26.495606 30010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:26.497215 30317 tablet_replica.cc:330] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: stopping tablet replica
I20260929 02:58:26.497567 30318 tablet_replica.cc:330] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: stopping tablet replica
I20260929 02:58:26.498034 30317 raft_consensus.cc:2229] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:58:26.498030 30318 raft_consensus.cc:2229] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:58:26.498595 30318 raft_consensus.cc:2258] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:26.498698 30317 raft_consensus.cc:2258] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:26.501185 30318 ts_tablet_manager.cc:1905] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:26.501184 30317 ts_tablet_manager.cc:1905] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:26.508689 30318 ts_tablet_manager.cc:1918] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:26.508872 30318 log.cc:1196] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/db4505e0c65c44f28a778b2c3d92e6eb
I20260929 02:58:26.509315 30317 ts_tablet_manager.cc:1918] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:26.509495 30318 ts_tablet_manager.cc:1939] T db4505e0c65c44f28a778b2c3d92e6eb P 36d543d2933b448499d3117db7ef3e8d: Deleting consensus metadata
I20260929 02:58:26.509724 30317 log.cc:1196] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/db4505e0c65c44f28a778b2c3d92e6eb
I20260929 02:58:26.510462 30317 ts_tablet_manager.cc:1939] T db4505e0c65c44f28a778b2c3d92e6eb P 58b578abe8e54b2c984de01880f27307: Deleting consensus metadata
I20260929 02:58:26.511355 29817 catalog_manager.cc:4841] TS 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763): tablet db4505e0c65c44f28a778b2c3d92e6eb (table test-table [id=d959a74b5fe2417782b8c84073a3eac3]) successfully deleted
I20260929 02:58:26.512091 29815 catalog_manager.cc:4841] TS 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743): tablet db4505e0c65c44f28a778b2c3d92e6eb (table test-table [id=d959a74b5fe2417782b8c84073a3eac3]) successfully deleted
I20260929 02:58:26.612053 30319 tablet_replica.cc:330] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: stopping tablet replica
I20260929 02:58:26.612452 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 29867
W20260929 02:58:26.626170 29817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.18.65:41655 (error 108)
I20260929 02:58:26.626863 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 30019
I20260929 02:58:26.628041 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.627573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41655 (local address 127.0.0.1:47846)
0929 02:58:26.627759 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.627874 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:58:26.628392 29817 catalog_manager.cc:4584] TS bc522fe93c2142dbb5b620d031653bd8 (127.15.18.65:41655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db4505e0c65c44f28a778b2c3d92e6eb: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41655: connect: Connection refused (error 111)
I20260929 02:58:26.643409 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.627637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.15.18.66:44763)
0929 02:58:26.627843 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.627848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.627950 (+ 102us) server_negotiation.cc:407] Connection header received
0929 02:58:26.628139 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.628145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.628226 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.628327 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:26.629602 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.630332 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.631566 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.631961 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.632052 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.638412 (+ 6360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:26.638427 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:26.638442 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:26.638480 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:26.641188 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:26.641802 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:26.641809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:26.641812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:26.641923 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:26.642215 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:26.642220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:26.642223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:26.642564 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:26.642790 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.642950 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.643107 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 02:58:26.644089 30142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:26.682010 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.681503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41655 (local address 127.0.0.1:47856)
0929 02:58:26.681737 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.681854 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 02:58:26.682966 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.682624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41655 (local address 127.0.0.1:47858)
0929 02:58:26.682771 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.682841 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 02:58:26.731940 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 30019
I20260929 02:58:26.743664 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.743144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41655 (local address 127.0.0.1:47874)
0929 02:58:26.743372 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.743462 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:58:26.744625 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.744340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41655 (local address 127.0.0.1:47890)
0929 02:58:26.744458 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.744524 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 02:58:26.745697 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 30150
I20260929 02:58:26.757798 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.746334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.15.18.67:46743)
0929 02:58:26.746521 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:26.746525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:26.746542 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:58:26.746642 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:26.746645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:26.746723 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:26.746800 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:26.748077 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.748809 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.750016 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:26.750406 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:26.750491 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:26.752874 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:26.752889 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:26.752904 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:26.752935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:26.755471 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:26.756130 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:26.756138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:26.756140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:26.756267 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:26.756546 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:26.756549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:26.756551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:26.756898 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:26.757099 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:26.757308 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 02:58:26.757462 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:58:26.758443 30273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:26.824846 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.824329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41655 (local address 127.0.0.1:47902)
0929 02:58:26.824542 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.824659 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 02:58:26.825985 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:26.825593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41655 (local address 127.0.0.1:47916)
0929 02:58:26.825800 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:26.825880 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 02:58:26.845609 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 30150
I20260929 02:58:26.858693 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 29797
I20260929 02:58:26.859810 29860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:26.971727 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 29797
I20260929 02:58:26.988644 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46199
--webserver_interface=127.15.18.126
--webserver_port=38799
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:46199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:27.303249 30325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:27.303582 30325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:27.303696 30325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:27.310864 30325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:58:27.310969 30325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:27.311014 30325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:27.311069 30325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:58:27.311112 30325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:58:27.321141 30325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:46199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:46199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=38799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:27.322399 30325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:27.323694 30325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:27.334880 30331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:27.334918 30330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:27.334942 30333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:27.335436 30325 server_base.cc:1034] running on GCE node
I20260929 02:58:27.336279 30325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:27.337636 30325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:27.338835 30325 hybrid_clock.cc:648] HybridClock initialized: now 1790650707338790 us; error 33 us; skew 500 ppm
I20260929 02:58:27.339154 30325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:27 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650707, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:27.346747 30325 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:58:27.348300 30325 webserver.cc:460] Webserver started at http://127.15.18.126:38799/ using document root <none> and password file <none>
I20260929 02:58:27.349175 30325 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:27.349347 30325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:27.354827 30325 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:58:27.357683 30340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:27.359031 30325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:58:27.359198 30325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2"
format_stamp: "Formatted at 2026-09-29 02:58:22 on dist-test-slave-q5f0"
server_key: "ea7bca598c70ed56c2826ea4b3a8ca03"
server_key_iv: "3c42297396471ca50d0456b194569cb1"
server_key_version: "encryptionkey@0"
I20260929 02:58:27.360270 30325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:27.382544 30325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:27.386543 30325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:27.386794 30325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:27.404299 30325 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:46199
I20260929 02:58:27.404327 30391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:46199 every 8 connection(s)
I20260929 02:58:27.405761 30325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:58:27.408609 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 30325
I20260929 02:58:27.409597 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:41655
--local_ip_for_outbound_sockets=127.15.18.65
--tserver_master_addrs=127.15.18.126:46199
--webserver_port=38253
--webserver_interface=127.15.18.65
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:58:27.414486 30392 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:58:27.418756 30392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Bootstrap starting.
I20260929 02:58:27.441272 30392 log.cc:824] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:27.456977 30392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:58:27.457536 30392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2: Bootstrap complete.
I20260929 02:58:27.466986 30392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:27.467929 30392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbd75b7ca1d49d1923a60a4a32a86d2, State: Initialized, Role: FOLLOWER
I20260929 02:58:27.468490 30392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:27.468695 30392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:58:27.468816 30392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:58:27.468961 30392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:58:27.472802 30392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:27.473286 30392 leader_election.cc:302] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dbd75b7ca1d49d1923a60a4a32a86d2; no voters:
I20260929 02:58:27.473873 30392 leader_election.cc:288] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 02:58:27.474059 30396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:58:27.474902 30396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [term 3 LEADER]: Becoming Leader. State: Replica: 3dbd75b7ca1d49d1923a60a4a32a86d2, State: Running, Role: LEADER
I20260929 02:58:27.475493 30396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } }
I20260929 02:58:27.475746 30392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:58:27.477917 30397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } } }
I20260929 02:58:27.478260 30397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:27.483319 30398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dbd75b7ca1d49d1923a60a4a32a86d2. Latest consensus state: current_term: 3 leader_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbd75b7ca1d49d1923a60a4a32a86d2" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 46199 } } }
I20260929 02:58:27.484146 30398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dbd75b7ca1d49d1923a60a4a32a86d2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:27.489853 30401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:58:27.498925 30401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:58:27.500407 30401 catalog_manager.cc:1259] Loaded cluster ID: df23d9fa9027401fa1044656283d4818
I20260929 02:58:27.500558 30401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 02:58:27.502518 30401 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f92195909ad google::LogMessage::Fail() at ??:0
@ 0x7f921959388a google::LogMessage::SendToLog() at ??:0
@ 0x7f92195904de google::LogMessage::Flush() at ??:0
@ 0x7f9219591a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f922efc3512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f922efc0976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f922f048610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f922f048439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f921c1dc345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f921c1df91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f921c1df799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f921c1baeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f921f4946db start_thread at ??:0
@ 0x7f921842a61f clone at ??:0
W20260929 02:58:27.798957 30394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:27.799238 30394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:27.799360 30394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:27.806573 30394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:27.806679 30394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:27.806830 30394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:58:27.816422 30394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:41655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=38253
--tserver_master_addrs=127.15.18.126:46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:27.817643 30394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:27.818902 30394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:27.831157 30433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:27.831172 30436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:27.831099 30434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:27.831776 30394 server_base.cc:1034] running on GCE node
I20260929 02:58:27.832342 30394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:27.833180 30394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:27.834370 30394 hybrid_clock.cc:648] HybridClock initialized: now 1790650707834330 us; error 34 us; skew 500 ppm
I20260929 02:58:27.834643 30394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:27 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650707, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:27.837517 30394 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:58:27.838846 30394 webserver.cc:460] Webserver started at http://127.15.18.65:38253/ using document root <none> and password file <none>
I20260929 02:58:27.839458 30394 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:27.839600 30394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:27.844607 30394 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:58:27.847262 30443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:27.848543 30394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:58:27.848680 30394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "bc522fe93c2142dbb5b620d031653bd8"
format_stamp: "Formatted at 2026-09-29 02:58:22 on dist-test-slave-q5f0"
server_key: "cb9efd712faff4865d997ca4a504247f"
server_key_iv: "8483325a5d39ec22724f82e1d2ea829b"
server_key_version: "encryptionkey@0"
I20260929 02:58:27.849607 30394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:27.868700 30394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:27.872757 30394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:27.873000 30394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:27.873942 30394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:27.875718 30450 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 02:58:27.881089 30394 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 02:58:27.881191 30394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 02:58:27.881307 30394 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 02:58:27.884404 30394 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 02:58:27.884478 30394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260929 02:58:27.884918 30450 tablet_bootstrap.cc:492] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Bootstrap starting.
I20260929 02:58:27.911065 30450 log.cc:824] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:27.922055 30394 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:41655
I20260929 02:58:27.922087 30557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:41655 every 8 connection(s)
I20260929 02:58:27.923008 30450 tablet_bootstrap.cc:492] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:58:27.923293 30394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:58:27.923499 30450 tablet_bootstrap.cc:492] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Bootstrap complete.
I20260929 02:58:27.925009 30450 ts_tablet_manager.cc:1397] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.016s
I20260929 02:58:27.925601 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 30394
I20260929 02:58:27.926177 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:44763
--local_ip_for_outbound_sockets=127.15.18.66
--tserver_master_addrs=127.15.18.126:46199
--webserver_port=46361
--webserver_interface=127.15.18.66
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:58:27.933367 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:27.926096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:45143)
0929 02:58:27.931969 (+ 5873us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:27.932320 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5519,"thread_start_us":5306,"threads_started":1}
W20260929 02:58:27.934197 30558 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:46199 (0 consecutive failures): Network error: Failed to ping master at 127.15.18.126:46199: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
I20260929 02:58:27.935191 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:27.934779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:45417)
0929 02:58:27.934967 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:27.935052 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 02:58:27.934476 30450 raft_consensus.cc:348] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:27.935655 30450 raft_consensus.cc:729] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc522fe93c2142dbb5b620d031653bd8, State: Initialized, Role: FOLLOWER
I20260929 02:58:27.936159 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:27.935783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:37763)
0929 02:58:27.935914 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:27.936007 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 02:58:27.936436 30558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:58:27.936282 30450 consensus_queue.cc:260] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:27.937870 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:27.937511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:54243)
0929 02:58:27.937660 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:27.937741 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 02:58:27.937875 30450 ts_tablet_manager.cc:1428] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260929 02:58:27.938740 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:27.938446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:44571)
0929 02:58:27.938565 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:27.938635 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:58:28.239758 30561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:28.240033 30561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:28.240154 30561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:28.247556 30561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:28.247677 30561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:28.247830 30561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:58:28.257953 30561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:44763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=46361
--tserver_master_addrs=127.15.18.126:46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:28.259255 30561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:28.260577 30561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:28.273291 30567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:28.273245 30570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:28.273244 30568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:28.274160 30561 server_base.cc:1034] running on GCE node
I20260929 02:58:28.274770 30561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:28.275743 30561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:28.276930 30561 hybrid_clock.cc:648] HybridClock initialized: now 1790650708276884 us; error 40 us; skew 500 ppm
I20260929 02:58:28.277232 30561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:28 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650708, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:28.280284 30561 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:58:28.281780 30561 webserver.cc:460] Webserver started at http://127.15.18.66:46361/ using document root <none> and password file <none>
I20260929 02:58:28.282490 30561 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:28.282593 30561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:28.288045 30561 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:58:28.290845 30577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:28.292234 30561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:58:28.292390 30561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "36d543d2933b448499d3117db7ef3e8d"
format_stamp: "Formatted at 2026-09-29 02:58:23 on dist-test-slave-q5f0"
server_key: "65f91f5e7b69dbadcf8a71d0f9a97e0d"
server_key_iv: "7bc91859f0c9633e0b0bfd613403f927"
server_key_version: "encryptionkey@0"
I20260929 02:58:28.293349 30561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:28.308562 30561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:28.312495 30561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:28.312743 30561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:28.313699 30561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:28.315011 30561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:28.315115 30561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:28.315224 30561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:28.315296 30561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:28.345451 30561 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:44763
I20260929 02:58:28.345515 30689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:44763 every 8 connection(s)
I20260929 02:58:28.346892 30561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:58:28.349018 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 30561
I20260929 02:58:28.349627 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:46743
--local_ip_for_outbound_sockets=127.15.18.67
--tserver_master_addrs=127.15.18.126:46199
--webserver_port=42669
--webserver_interface=127.15.18.67
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:58:28.351897 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:28.349920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:42815)
0929 02:58:28.350687 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:28.350948 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":307,"threads_started":1}
W20260929 02:58:28.352550 30690 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:46199 (0 consecutive failures): Network error: Failed to ping master at 127.15.18.126:46199: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
I20260929 02:58:28.353404 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:28.353091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:43205)
0929 02:58:28.353233 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:28.353299 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 02:58:28.354212 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:28.353881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:48523)
0929 02:58:28.354024 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:28.354108 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 02:58:28.354418 30690 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:58:28.673092 30693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:28.673367 30693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:28.673484 30693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:28.680660 30693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:28.680770 30693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:28.680922 30693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:58:28.690512 30693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:46743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=42669
--tserver_master_addrs=127.15.18.126:46199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:28.691776 30693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:28.693030 30693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:28.705896 30699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:28.705963 30698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:28.705991 30701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:28.706544 30693 server_base.cc:1034] running on GCE node
I20260929 02:58:28.707180 30693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:28.708204 30693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:28.709403 30693 hybrid_clock.cc:648] HybridClock initialized: now 1790650708709357 us; error 38 us; skew 500 ppm
I20260929 02:58:28.709707 30693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:28 dist-test-slave-q5f0 krb5kdc[29247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650708, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:28.712692 30693 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:58:28.714071 30693 webserver.cc:460] Webserver started at http://127.15.18.67:42669/ using document root <none> and password file <none>
I20260929 02:58:28.714735 30693 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:28.714850 30693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:28.720044 30693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:58:28.722641 30708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:28.723980 30693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:58:28.724125 30693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "58b578abe8e54b2c984de01880f27307"
format_stamp: "Formatted at 2026-09-29 02:58:23 on dist-test-slave-q5f0"
server_key: "9d8367d4d00ec37abcf22ddc6ac52242"
server_key_iv: "1aeedf18244bdce454db02f74cc96f59"
server_key_version: "encryptionkey@0"
I20260929 02:58:28.725065 30693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:28.739714 30693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:28.743605 30693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:28.743841 30693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:28.744859 30693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:28.746168 30693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:28.746268 30693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:28.746366 30693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:28.746438 30693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:28.775905 30693 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:46743
I20260929 02:58:28.775971 30820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:46743 every 8 connection(s)
I20260929 02:58:28.777307 30693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:58:28.780647 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 30693
I20260929 02:58:28.782212 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:28.780109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:52071)
0929 02:58:28.780888 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:28.781094 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":313,"threads_started":1}
W20260929 02:58:28.782917 30821 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:46199 (0 consecutive failures): Network error: Failed to ping master at 127.15.18.126:46199: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
I20260929 02:58:28.783741 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:28.783348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:51397)
0929 02:58:28.783563 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:28.783635 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 02:58:28.784512 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:28.784209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:45743)
0929 02:58:28.784338 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:28.784407 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 02:58:28.784732 30821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:58:28.940234 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:28.939485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:60229)
0929 02:58:28.939950 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:28.940041 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":216,"threads_started":1}
I20260929 02:58:29.355890 30826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:29.355008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:55197)
0929 02:58:29.355501 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:29.355650 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":241,"threads_started":1}
I20260929 02:58:29.594038 30827 raft_consensus.cc:482] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:58:29.594274 30827 raft_consensus.cc:504] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:29.595805 30827 leader_election.cc:288] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743), 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763)
Sep 29 02:58:29 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650707, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
Sep 29 02:58:29 dist-test-slave-q5f0 krb5kdc[29247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650707, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:58:29.628561 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:29.596581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:56071 (local address 127.15.18.66:44763)
0929 02:58:29.607743 (+ 11162us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:29.607765 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:29.609045 (+ 1280us) server_negotiation.cc:407] Connection header received
0929 02:58:29.611177 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:29.611185 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:29.611583 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:29.611887 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:29.613996 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:29.615143 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:29.616962 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:29.617570 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:29.617730 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:29.621848 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:29.621857 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:29.621874 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:29.621918 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:29.625003 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:29.625815 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:29.625821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:29.625826 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:29.625974 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:29.626406 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:29.626410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:29.626412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:29.626887 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:29.627052 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:29.627830 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 02:58:29.628175 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10909,"thread_start_us":11213,"threads_started":1}
I20260929 02:58:29.628580 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:29.596128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46743 (local address 127.15.18.65:48109)
0929 02:58:29.609018 (+ 12890us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:29.609036 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:29.609490 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:29.612026 (+ 2536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:29.612036 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:29.613004 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:29.613822 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:29.613841 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:29.615318 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:29.615322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:29.616840 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:29.616852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:29.617762 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:29.617768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:29.618054 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:29.618823 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:29.618845 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:29.621285 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:29.625199 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:29.625203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:29.625205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:29.625678 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:29.626081 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:29.626085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:29.626087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:29.626280 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:29.627025 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:29.627032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:29.627326 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 02:58:29.627587 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 02:58:29.628098 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12768,"mutex_wait_us":237,"spinlock_wait_cycles":60800,"thread_start_us":12605,"threads_started":1}
I20260929 02:58:29.628679 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:29.596114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:44763 (local address 127.15.18.65:56071)
0929 02:58:29.608849 (+ 12735us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:29.608887 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:29.609582 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:29.612027 (+ 2445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:29.612037 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:29.612986 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:29.613821 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:29.613841 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:29.615319 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:29.615322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:29.616835 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:29.616847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:29.617807 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:29.617815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:29.618057 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:29.618787 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:29.618825 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:29.621565 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:29.625137 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:29.625146 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:29.625160 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:29.625671 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:29.626080 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:29.626085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:29.626089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:29.626296 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:29.627025 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:29.627032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:29.627326 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 02:58:29.627588 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 02:58:29.628246 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12614,"mutex_wait_us":438,"spinlock_wait_cycles":28928,"thread_start_us":12379,"threads_started":1}
I20260929 02:58:29.628665 30830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:29.596298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:48109 (local address 127.15.18.67:46743)
0929 02:58:29.608432 (+ 12134us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:29.608450 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:29.609157 (+ 707us) server_negotiation.cc:407] Connection header received
0929 02:58:29.611167 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:29.611176 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:29.611585 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:29.611887 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:29.613996 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:29.615162 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:29.616969 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:29.617567 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:29.617722 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:29.621582 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:29.621589 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:29.621604 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:29.621647 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:29.625071 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:29.625810 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:29.625817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:29.625823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:29.625976 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:29.626392 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:29.626396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:29.626400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:29.626887 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:29.627052 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:29.627843 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 02:58:29.628178 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11938,"thread_start_us":11829,"threads_started":1}
W20260929 02:58:29.631568 30445 leader_election.cc:341] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743): Not found: Tablet not found: db4505e0c65c44f28a778b2c3d92e6eb
W20260929 02:58:29.631910 30447 leader_election.cc:341] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763): Not found: Tablet not found: db4505e0c65c44f28a778b2c3d92e6eb
I20260929 02:58:29.632043 30447 leader_election.cc:302] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc522fe93c2142dbb5b620d031653bd8; no voters: 36d543d2933b448499d3117db7ef3e8d, 58b578abe8e54b2c984de01880f27307
I20260929 02:58:29.632433 30827 raft_consensus.cc:2726] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:58:29.786010 30832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:29.785237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:42991)
0929 02:58:29.785658 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:29.785793 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":193,"threads_started":1}
I20260929 02:58:29.941509 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:29.941083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:43851)
0929 02:58:29.941267 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:29.941364 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 02:58:30.361867 30833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:30.361055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:46809)
0929 02:58:30.361574 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:30.361661 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":288,"threads_started":1}
I20260929 02:58:30.787580 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:30.786867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:55653)
0929 02:58:30.787306 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:30.787387 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":234,"threads_started":1}
I20260929 02:58:30.943225 30835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:30.942385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:47337)
0929 02:58:30.942928 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:30.943030 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":242,"threads_started":1}
I20260929 02:58:31.363492 30836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:31.362734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:48023)
0929 02:58:31.363220 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:31.363318 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":233,"threads_started":1}
I20260929 02:58:31.710836 30837 raft_consensus.cc:482] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:58:31.711025 30837 raft_consensus.cc:504] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:31.711699 30837 leader_election.cc:288] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743), 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763)
W20260929 02:58:31.712769 30447 leader_election.cc:341] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763): Not found: Tablet not found: db4505e0c65c44f28a778b2c3d92e6eb
W20260929 02:58:31.712956 30445 leader_election.cc:341] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743): Not found: Tablet not found: db4505e0c65c44f28a778b2c3d92e6eb
I20260929 02:58:31.713076 30445 leader_election.cc:302] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc522fe93c2142dbb5b620d031653bd8; no voters: 36d543d2933b448499d3117db7ef3e8d, 58b578abe8e54b2c984de01880f27307
I20260929 02:58:31.713332 30837 raft_consensus.cc:2726] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:58:31.789103 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:31.788383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:44663)
0929 02:58:31.788843 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:31.788930 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":238,"threads_started":1}
I20260929 02:58:31.944931 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:31.944143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:43825)
0929 02:58:31.944635 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:31.944754 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":218,"threads_started":1}
I20260929 02:58:32.365113 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:32.364398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:34969)
0929 02:58:32.364831 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:32.364913 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260929 02:58:32.790742 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:32.789989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:46349)
0929 02:58:32.790448 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:32.790534 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":224,"threads_started":1}
I20260929 02:58:32.946870 30842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:32.946025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:45579)
0929 02:58:32.946568 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:32.946671 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":289,"threads_started":1}
I20260929 02:58:33.366855 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:33.366108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:57849)
0929 02:58:33.366590 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:33.366680 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":261,"threads_started":1}
I20260929 02:58:33.792568 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:33.791711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:33343)
0929 02:58:33.792197 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:33.792280 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":261,"threads_started":1}
I20260929 02:58:33.948571 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:33.947816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:40691)
0929 02:58:33.948288 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:33.948368 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":275,"threads_started":1}
I20260929 02:58:34.368512 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:34.367789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:35045)
0929 02:58:34.368247 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:34.368331 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":232,"threads_started":1}
I20260929 02:58:34.794226 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:34.793462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:41943)
0929 02:58:34.793963 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:34.794057 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":253,"threads_started":1}
I20260929 02:58:34.950405 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:34.949504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:51661)
0929 02:58:34.950045 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:34.950165 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":320,"threads_started":1}
I20260929 02:58:35.014721 30849 raft_consensus.cc:482] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:58:35.014919 30849 raft_consensus.cc:504] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:35.015595 30849 leader_election.cc:288] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743), 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763)
W20260929 02:58:35.016448 30445 leader_election.cc:341] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743): Not found: Tablet not found: db4505e0c65c44f28a778b2c3d92e6eb
W20260929 02:58:35.016610 30447 leader_election.cc:341] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763): Not found: Tablet not found: db4505e0c65c44f28a778b2c3d92e6eb
I20260929 02:58:35.016758 30447 leader_election.cc:302] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc522fe93c2142dbb5b620d031653bd8; no voters: 36d543d2933b448499d3117db7ef3e8d, 58b578abe8e54b2c984de01880f27307
I20260929 02:58:35.017001 30849 raft_consensus.cc:2726] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:58:35.370293 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:35.369456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:40863)
0929 02:58:35.370008 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:35.370106 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":258,"threads_started":1}
I20260929 02:58:35.796108 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:35.795229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:46699)
0929 02:58:35.795775 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:35.795871 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
I20260929 02:58:35.952126 30852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:35.951327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:32863)
0929 02:58:35.951848 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:35.951947 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":274,"threads_started":1}
I20260929 02:58:36.372016 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:36.371275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:50183)
0929 02:58:36.371743 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:36.371833 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":251,"threads_started":1}
I20260929 02:58:36.797763 30854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:36.797003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:36613)
0929 02:58:36.797480 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:36.797569 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":254,"threads_started":1}
I20260929 02:58:36.953783 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:36.953005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:33369)
0929 02:58:36.953497 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:36.953596 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":285,"threads_started":1}
I20260929 02:58:37.373728 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:37.372951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:42519)
0929 02:58:37.373442 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:37.373535 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":231,"threads_started":1}
I20260929 02:58:37.799366 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:37.798630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:55233)
0929 02:58:37.799109 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:37.799198 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":222,"threads_started":1}
I20260929 02:58:37.955401 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:37.954692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:56107)
0929 02:58:37.955147 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:37.955233 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":225,"threads_started":1}
I20260929 02:58:38.375416 30859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:38.374677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:54133)
0929 02:58:38.375133 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:38.375220 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
I20260929 02:58:38.800956 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:38.800246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:44165)
0929 02:58:38.800682 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:38.800763 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":210,"threads_started":1}
I20260929 02:58:38.956846 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:38.956200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:47567)
0929 02:58:38.956587 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:38.956668 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":203,"threads_started":1}
I20260929 02:58:39.377125 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:39.376349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:42629)
0929 02:58:39.376869 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:39.376954 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":263,"threads_started":1}
I20260929 02:58:39.802526 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:39.801804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:52649)
0929 02:58:39.802271 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:39.802359 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":223,"threads_started":1}
I20260929 02:58:39.920367 30864 raft_consensus.cc:482] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:58:39.920603 30864 raft_consensus.cc:504] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58b578abe8e54b2c984de01880f27307" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46743 } } peers { permanent_uuid: "36d543d2933b448499d3117db7ef3e8d" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 44763 } } peers { permanent_uuid: "bc522fe93c2142dbb5b620d031653bd8" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41655 } }
I20260929 02:58:39.921420 30864 leader_election.cc:288] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743), 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763)
W20260929 02:58:39.922364 30445 leader_election.cc:341] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58b578abe8e54b2c984de01880f27307 (127.15.18.67:46743): Not found: Tablet not found: db4505e0c65c44f28a778b2c3d92e6eb
W20260929 02:58:39.922547 30447 leader_election.cc:341] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36d543d2933b448499d3117db7ef3e8d (127.15.18.66:44763): Not found: Tablet not found: db4505e0c65c44f28a778b2c3d92e6eb
I20260929 02:58:39.922698 30447 leader_election.cc:302] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc522fe93c2142dbb5b620d031653bd8; no voters: 36d543d2933b448499d3117db7ef3e8d, 58b578abe8e54b2c984de01880f27307
I20260929 02:58:39.922952 30864 raft_consensus.cc:2726] T db4505e0c65c44f28a778b2c3d92e6eb P bc522fe93c2142dbb5b620d031653bd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:58:39.958495 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:39.957668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:47351)
0929 02:58:39.958167 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:39.958281 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":237,"threads_started":1}
I20260929 02:58:40.378851 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:40.378100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:41713)
0929 02:58:40.378554 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:40.378676 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":228,"threads_started":1}
I20260929 02:58:40.804133 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:40.803372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:58007)
0929 02:58:40.803867 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:40.803950 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":271,"threads_started":1}
I20260929 02:58:40.960139 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:40.959338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:58937)
0929 02:58:40.959857 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:40.959945 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":286,"threads_started":1}
I20260929 02:58:41.380707 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:41.379901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:54855)
0929 02:58:41.380422 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:41.380525 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":254,"threads_started":1}
I20260929 02:58:41.805725 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:41.805015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:46223)
0929 02:58:41.805469 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:41.805557 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
I20260929 02:58:41.961759 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:41.960976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:37969)
0929 02:58:41.961491 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:41.961575 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":286,"threads_started":1}
I20260929 02:58:42.382288 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:42.381508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:41347)
0929 02:58:42.382004 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:42.382082 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":282,"threads_started":1}
I20260929 02:58:42.807432 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:42.806673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:50215)
0929 02:58:42.807147 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:42.807228 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":246,"threads_started":1}
I20260929 02:58:42.963721 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:42.962935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.65:40251)
0929 02:58:42.963405 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:42.963498 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
I20260929 02:58:43.384159 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:43.383349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.66:36369)
0929 02:58:43.383880 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:43.383998 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":293,"threads_started":1}
I20260929 02:58:43.782121 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 30394
I20260929 02:58:43.802330 30876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:43.783186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.15.18.65:41655)
0929 02:58:43.783808 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:43.783815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:43.783841 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:58:43.783901 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:43.783908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:43.784116 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:43.784307 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:43.786357 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:43.787328 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:43.788727 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:43.789364 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:43.789494 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:43.796713 (+ 7219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:43.796727 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:43.796743 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:43.796783 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:43.799641 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:43.800324 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:43.800332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:43.800334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:43.800462 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:43.800839 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:43.800842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:43.800844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:43.801191 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:43.801479 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:43.801768 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 02:58:43.802023 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":267,"threads_started":1}
I20260929 02:58:43.803617 30553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:43.809114 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:43.808422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:46199 (local address 127.15.18.67:40229)
0929 02:58:43.808852 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:43.808972 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:46199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":217,"threads_started":1}
I20260929 02:58:43.891163 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 30394
I20260929 02:58:43.905237 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 30561
I20260929 02:58:43.918171 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:43.905961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.15.18.66:44763)
0929 02:58:43.906448 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:43.906452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:43.906467 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:58:43.906520 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:43.906523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:43.906580 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:43.906709 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:43.908121 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:43.908899 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:43.910105 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:43.910527 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:43.910635 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:43.913159 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:43.913164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:43.913168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:43.913198 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:43.915696 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:43.916341 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:43.916347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:43.916350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:43.916468 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:43.916804 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:43.916809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:43.916813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:43.917079 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:43.917333 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:43.917586 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 02:58:43.917809 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":245,"threads_started":1}
I20260929 02:58:43.918896 30685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:44.001235 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 30561
I20260929 02:58:44.015096 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 30693
I20260929 02:58:44.028116 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:44.016051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.15.18.67:46743)
0929 02:58:44.016647 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:44.016652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:44.016669 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:58:44.016721 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:44.016725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:44.016791 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:44.016876 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:44.018254 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:44.019036 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:44.020435 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:44.020864 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:44.020962 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:44.023332 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:44.023337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:44.023340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:44.023371 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:44.025861 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:44.026525 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:44.026529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:44.026531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:44.026655 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:44.026927 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:44.026930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:44.026931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:44.027159 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:44.027358 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:44.027605 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 02:58:44.027799 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":291,"threads_started":1}
I20260929 02:58:44.028810 30816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:44.111963 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 30693
2026-09-29T02:58:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23563 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:58:44 dist-test-slave-q5f0 krb5kdc[30885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:58:44 dist-test-slave-q5f0 krb5kdc[30885](info): set up 2 sockets
Sep 29 02:58:44 dist-test-slave-q5f0 krb5kdc[30885](info): commencing operation
krb5kdc: starting...
W20260929 02:58:45.128327 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:58:45 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:58:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:58:45.351554 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:34067
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:45.668147 30898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:45.668440 30898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:45.668565 30898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:45.675669 30898 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:58:45.675781 30898 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:58:45.675835 30898 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:58:45.675904 30898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:58:45.675976 30898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:45.676067 30898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:45.676131 30898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:58:45.676204 30898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:58:45.685964 30898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:34067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:45.687229 30898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:45.688575 30898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:45.699971 30903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:45.700053 30906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:45.700060 30904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:45.700644 30898 server_base.cc:1034] running on GCE node
I20260929 02:58:45.701422 30898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:45.702802 30898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:45.704012 30898 hybrid_clock.cc:648] HybridClock initialized: now 1790650725703950 us; error 35 us; skew 500 ppm
I20260929 02:58:45.704330 30898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:45 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650725, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:45.707211 30898 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:58:45.708652 30898 webserver.cc:460] Webserver started at http://127.15.18.126:39987/ using document root <none> and password file <none>
I20260929 02:58:45.709331 30898 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:45.709427 30898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:45.709712 30898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:45.712648 30898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "9997807127644aa6b091762cc2d0f82a"
format_stamp: "Formatted at 2026-09-29 02:58:45 on dist-test-slave-q5f0"
server_key: "099f5b19c7b3226d6fa5f4932b049fe7"
server_key_iv: "d191dbe36babe2fc1bc49306313afc79"
server_key_version: "encryptionkey@0"
I20260929 02:58:45.713374 30898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "9997807127644aa6b091762cc2d0f82a"
format_stamp: "Formatted at 2026-09-29 02:58:45 on dist-test-slave-q5f0"
server_key: "099f5b19c7b3226d6fa5f4932b049fe7"
server_key_iv: "d191dbe36babe2fc1bc49306313afc79"
server_key_version: "encryptionkey@0"
I20260929 02:58:45.718300 30898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:58:45.721710 30913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:45.722941 30898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 02:58:45.723086 30898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "9997807127644aa6b091762cc2d0f82a"
format_stamp: "Formatted at 2026-09-29 02:58:45 on dist-test-slave-q5f0"
server_key: "099f5b19c7b3226d6fa5f4932b049fe7"
server_key_iv: "d191dbe36babe2fc1bc49306313afc79"
server_key_version: "encryptionkey@0"
I20260929 02:58:45.723292 30898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:45.748308 30898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:45.752231 30898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:45.752472 30898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:45.769874 30898 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:34067
I20260929 02:58:45.769896 30964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:34067 every 8 connection(s)
I20260929 02:58:45.771226 30898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:58:45.772244 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 30898
I20260929 02:58:45.772482 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:58:45.773270 15433 external_mini_cluster.cc:1428] Setting key 23b57133ed990847458fdeb9012eb5cd
I20260929 02:58:45.775638 30965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:58:45 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:45.789011 30965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Bootstrap starting.
I20260929 02:58:45.792399 30965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:45.793576 30965 log.cc:824] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:45.796811 30965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: No bootstrap required, opened a new log
I20260929 02:58:45.796849 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:45.775442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.15.18.126:34067)
0929 02:58:45.776383 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:45.776395 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:45.776435 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:58:45.778072 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:45.778098 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:45.778562 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:45.778962 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:45.780792 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:45.782158 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:45.784026 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:45.784728 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:45.784865 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:45.789670 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:45.789678 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:45.789697 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:45.789736 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:45.792630 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:45.793423 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:45.793429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:45.793434 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:45.793593 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:45.794035 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:45.794040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:45.794043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:45.794813 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:45.795108 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:45.795492 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 02:58:45.795879 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":453,"threads_started":1}
I20260929 02:58:45.803885 30965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:58:45.804224 30965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:45.804322 30965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9997807127644aa6b091762cc2d0f82a, State: Initialized, Role: FOLLOWER
I20260929 02:58:45.804837 30965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:58:45.805030 30965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:58:45.805158 30965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:58:45.805320 30965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:45.807416 30965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:58:45.807977 30965 leader_election.cc:302] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9997807127644aa6b091762cc2d0f82a; no voters:
I20260929 02:58:45.808521 30965 leader_election.cc:288] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:58:45.808656 30970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:58:45.809754 30970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 1 LEADER]: Becoming Leader. State: Replica: 9997807127644aa6b091762cc2d0f82a, State: Running, Role: LEADER
I20260929 02:58:45.810307 30970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:58:45.810664 30965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:58:45.813056 30971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9997807127644aa6b091762cc2d0f82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } } }
I20260929 02:58:45.813151 30972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9997807127644aa6b091762cc2d0f82a. Latest consensus state: current_term: 1 leader_uuid: "9997807127644aa6b091762cc2d0f82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } } }
I20260929 02:58:45.813462 30971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:45.813479 30972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: This master's current role is: LEADER
I20260929 02:58:45.814296 30978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:58:45.820088 30978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:58:45.829066 30978 catalog_manager.cc:1347] Generated new cluster ID: 039f016e23fc484fa18a28ad0ff39b97
I20260929 02:58:45.829180 30978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:58:45.841100 30978 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:58:45.842919 30978 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:58:45.850368 30978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Generated new TSK 0
I20260929 02:58:45.851125 30978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:58:45.911882 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34067
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:46.225272 30991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:46.225540 30991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:46.225641 30991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:46.233139 30991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:46.233253 30991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:46.233374 30991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:58:46.243144 30991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:46.244407 30991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:46.245675 30991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:46.258311 30996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:46.258265 30999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:46.258225 30997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:46.259076 30991 server_base.cc:1034] running on GCE node
I20260929 02:58:46.259709 30991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:46.260638 30991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:46.261853 30991 hybrid_clock.cc:648] HybridClock initialized: now 1790650726261794 us; error 51 us; skew 500 ppm
I20260929 02:58:46.262190 30991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:46 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650726, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:46.265203 30991 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:58:46.266577 30991 webserver.cc:460] Webserver started at http://127.15.18.65:46869/ using document root <none> and password file <none>
I20260929 02:58:46.267223 30991 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:46.267339 30991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:46.267669 30991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:46.270573 30991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "cb465fabbbe54e20a59b2c7daba3e538"
format_stamp: "Formatted at 2026-09-29 02:58:46 on dist-test-slave-q5f0"
server_key: "9655e6851ee33470683e906ca5ac65d5"
server_key_iv: "6636ac715242fab654e3f03cd9d0e850"
server_key_version: "encryptionkey@0"
I20260929 02:58:46.271322 30991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "cb465fabbbe54e20a59b2c7daba3e538"
format_stamp: "Formatted at 2026-09-29 02:58:46 on dist-test-slave-q5f0"
server_key: "9655e6851ee33470683e906ca5ac65d5"
server_key_iv: "6636ac715242fab654e3f03cd9d0e850"
server_key_version: "encryptionkey@0"
I20260929 02:58:46.276520 30991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:58:46.280030 31006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:46.281293 30991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:58:46.281437 30991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "cb465fabbbe54e20a59b2c7daba3e538"
format_stamp: "Formatted at 2026-09-29 02:58:46 on dist-test-slave-q5f0"
server_key: "9655e6851ee33470683e906ca5ac65d5"
server_key_iv: "6636ac715242fab654e3f03cd9d0e850"
server_key_version: "encryptionkey@0"
I20260929 02:58:46.281622 30991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:46.305519 30991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:46.310035 30991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:46.310356 30991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:46.311465 30991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:46.313015 30991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:46.313102 30991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:46.313231 30991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:46.313288 30991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:46.344746 30991 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:41293
I20260929 02:58:46.344789 31118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:41293 every 8 connection(s)
I20260929 02:58:46.346076 30991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:58:46.353881 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 30991
I20260929 02:58:46.354081 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:58:46.354840 15433 external_mini_cluster.cc:1428] Setting key bc7fccaf34c91e5a4214ba468f864fff
Sep 29 02:58:46 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650726, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:46.365377 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:46.348528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:55103 (local address 127.15.18.126:34067)
0929 02:58:46.348978 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:46.348984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:46.349930 (+ 946us) server_negotiation.cc:407] Connection header received
0929 02:58:46.351916 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:46.351920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:46.351984 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:46.352075 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:46.353579 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:46.354605 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:46.356223 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:46.356613 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:46.356723 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:46.360140 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:46.360145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:46.360149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:46.360183 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:46.362484 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:46.363156 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:46.363160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:46.363162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:46.363279 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:46.363731 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:46.363734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:46.363736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:46.363995 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:46.364125 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:46.364810 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 02:58:46.365028 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":211,"threads_started":1}
I20260929 02:58:46.366335 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:46.349039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:55103)
0929 02:58:46.349763 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:46.349811 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:46.351673 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:46.352333 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:46.352341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:46.352859 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:46.353403 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:46.353420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:46.354751 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:46.354755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:46.356069 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:46.356081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:46.356826 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:46.356832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:46.357037 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:46.357653 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:46.357694 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:46.359931 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:46.362613 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:46.362622 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:46.362639 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:46.363036 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:46.363378 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:46.363383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:46.363387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:46.363619 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:46.364121 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:46.364128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:46.364449 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 02:58:46.364672 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 02:58:46.365198 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":282,"threads_started":1}
I20260929 02:58:46.367841 31119 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34067
I20260929 02:58:46.368152 31119 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:46.369035 31119 heartbeater.cc:511] Master 127.15.18.126:34067 requested a full tablet report, sending...
I20260929 02:58:46.371431 30930 ts_manager.cc:194] Registered new tserver with Master: cb465fabbbe54e20a59b2c7daba3e538 (127.15.18.65:41293)
I20260929 02:58:46.373459 30930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:55103
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:58:46.403121 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34067
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:46.708910 31125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:46.709165 31125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:46.709256 31125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:46.716369 31125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:46.716465 31125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:46.716607 31125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:58:46.726075 31125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:46.727298 31125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:46.728595 31125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:46.740664 31130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:46.740700 31131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:46.740736 31133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:46.741187 31125 server_base.cc:1034] running on GCE node
I20260929 02:58:46.741765 31125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:46.742632 31125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:46.743817 31125 hybrid_clock.cc:648] HybridClock initialized: now 1790650726743771 us; error 39 us; skew 500 ppm
I20260929 02:58:46.744102 31125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:46 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650726, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:46.747011 31125 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:58:46.748370 31125 webserver.cc:460] Webserver started at http://127.15.18.66:33499/ using document root <none> and password file <none>
I20260929 02:58:46.748996 31125 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:46.749104 31125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:46.749398 31125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:46.752202 31125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "e0e72a441d1b45769ff4341eeb4be6a1"
format_stamp: "Formatted at 2026-09-29 02:58:46 on dist-test-slave-q5f0"
server_key: "6358f5e6bf70708f01b81f1f8515bacc"
server_key_iv: "f6dc3eef243486c90a808453a4bc0bca"
server_key_version: "encryptionkey@0"
I20260929 02:58:46.752905 31125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "e0e72a441d1b45769ff4341eeb4be6a1"
format_stamp: "Formatted at 2026-09-29 02:58:46 on dist-test-slave-q5f0"
server_key: "6358f5e6bf70708f01b81f1f8515bacc"
server_key_iv: "f6dc3eef243486c90a808453a4bc0bca"
server_key_version: "encryptionkey@0"
I20260929 02:58:46.758119 31125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:58:46.761497 31140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:46.762761 31125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:58:46.762894 31125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "e0e72a441d1b45769ff4341eeb4be6a1"
format_stamp: "Formatted at 2026-09-29 02:58:46 on dist-test-slave-q5f0"
server_key: "6358f5e6bf70708f01b81f1f8515bacc"
server_key_iv: "f6dc3eef243486c90a808453a4bc0bca"
server_key_version: "encryptionkey@0"
I20260929 02:58:46.763082 31125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:46.775202 31125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:46.779142 31125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:46.779429 31125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:46.780467 31125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:46.781800 31125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:46.781875 31125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:46.781987 31125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:46.782037 31125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:46.810848 31125 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:39415
I20260929 02:58:46.810885 31252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:39415 every 8 connection(s)
I20260929 02:58:46.812172 31125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:58:46.814114 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 31125
I20260929 02:58:46.814283 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:58:46.814847 15433 external_mini_cluster.cc:1428] Setting key 4972dfcc955a5aa52b923535af3f90e6
Sep 29 02:58:46 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650726, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:46.831602 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:46.814549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:43649 (local address 127.15.18.126:34067)
0929 02:58:46.815461 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:46.815467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:46.816027 (+ 560us) server_negotiation.cc:407] Connection header received
0929 02:58:46.817827 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:46.817833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:46.817913 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:46.818000 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:46.819488 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:46.820438 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:46.822052 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:46.822501 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:46.822612 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:46.826103 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:46.826108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:46.826111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:46.826142 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:46.828681 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:46.829463 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:46.829466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:46.829468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:46.829584 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:46.829998 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:46.830001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:46.830002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:46.830249 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:46.830360 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:46.831149 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 02:58:46.831318 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767}
I20260929 02:58:46.833017 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:46.815038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:43649)
0929 02:58:46.815863 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:46.815920 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:46.817560 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:46.818241 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:46.818249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:46.818757 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:46.819326 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:46.819344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:46.820583 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:46.820587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:46.821916 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:46.821928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:46.822714 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:46.822721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:46.822931 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:46.823559 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:46.823582 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:46.825879 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:46.828812 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:46.828821 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:46.828838 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:46.829370 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:46.829680 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:46.829684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:46.829688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:46.829908 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:46.830355 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:46.830364 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:46.830790 (+ 426us) client_negotiation.cc:769] Sending connection context
0929 02:58:46.831081 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 02:58:46.831619 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":308,"threads_started":1}
I20260929 02:58:46.834380 31253 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34067
I20260929 02:58:46.834725 31253 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:46.835639 31253 heartbeater.cc:511] Master 127.15.18.126:34067 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
I20260929 02:58:46.837200 30930 ts_manager.cc:194] Registered new tserver with Master: e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415)
I20260929 02:58:46.837999 30930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:43649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:58:46.862632 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34067
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:58:47.183647 31258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:58:47.183964 31258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:58:47.184137 31258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:58:47.191603 31258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:58:47.191747 31258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:58:47.191918 31258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:58:47.201916 31258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:58:47.203290 31258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:58:47.204656 31258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:58:47.218017 31263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:47.218051 31266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:58:47.217975 31264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:58:47.218858 31258 server_base.cc:1034] running on GCE node
I20260929 02:58:47.219442 31258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:58:47.220427 31258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:58:47.221658 31258 hybrid_clock.cc:648] HybridClock initialized: now 1790650727221606 us; error 34 us; skew 500 ppm
I20260929 02:58:47.222007 31258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:58:47 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650727, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:58:47.225199 31258 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:58:47.226696 31258 webserver.cc:460] Webserver started at http://127.15.18.67:33767/ using document root <none> and password file <none>
I20260929 02:58:47.227373 31258 fs_manager.cc:359] Metadata directory not provided
I20260929 02:58:47.227473 31258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:58:47.227814 31258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:58:47.230726 31258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "1a31d48ec1bc41dc8766d469412bfef3"
format_stamp: "Formatted at 2026-09-29 02:58:47 on dist-test-slave-q5f0"
server_key: "801833e89d12ae83d5f8212019f8bcfc"
server_key_iv: "0359150d0df66a7337616fe8d0465669"
server_key_version: "encryptionkey@0"
I20260929 02:58:47.231446 31258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "1a31d48ec1bc41dc8766d469412bfef3"
format_stamp: "Formatted at 2026-09-29 02:58:47 on dist-test-slave-q5f0"
server_key: "801833e89d12ae83d5f8212019f8bcfc"
server_key_iv: "0359150d0df66a7337616fe8d0465669"
server_key_version: "encryptionkey@0"
I20260929 02:58:47.236577 31258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:58:47.240049 31273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:47.241441 31258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:58:47.241647 31258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "1a31d48ec1bc41dc8766d469412bfef3"
format_stamp: "Formatted at 2026-09-29 02:58:47 on dist-test-slave-q5f0"
server_key: "801833e89d12ae83d5f8212019f8bcfc"
server_key_iv: "0359150d0df66a7337616fe8d0465669"
server_key_version: "encryptionkey@0"
I20260929 02:58:47.241852 31258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:58:47.257845 31258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:58:47.262372 31258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:58:47.262684 31258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:58:47.263689 31258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:58:47.265060 31258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:58:47.265141 31258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:47.265246 31258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:58:47.265311 31258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:58:47.294207 31258 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:33835
I20260929 02:58:47.294238 31385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:33835 every 8 connection(s)
I20260929 02:58:47.295624 31258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:58:47.302534 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 31258
I20260929 02:58:47.302736 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:58:47.303258 15433 external_mini_cluster.cc:1428] Setting key aa3219c2b73884a9ffd20b0a33d296d6
Sep 29 02:58:47 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650727, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:58:47.315274 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.298228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:41683 (local address 127.15.18.126:34067)
0929 02:58:47.298410 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.298415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.299555 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 02:58:47.301316 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.301320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.301390 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.301476 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:47.303157 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.304045 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.305913 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.306498 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.306615 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.309847 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:47.309853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:47.309856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:47.309888 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:47.312490 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:47.313153 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:47.313157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:47.313160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:47.313279 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:47.313662 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:47.313666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:47.313670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:47.313922 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:47.314050 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.314694 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.314904 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:58:47.316283 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.298708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:41683)
0929 02:58:47.299384 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:47.299421 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:47.301052 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:47.301673 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:47.301681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:47.302212 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:58:47.302986 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.303009 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.304203 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.304208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.305762 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.305773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.306712 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.306718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.306911 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.307532 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:58:47.307563 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:58:47.309656 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:58:47.312613 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:47.312623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:47.312640 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:47.313045 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:47.313377 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:58:47.313382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:58:47.313386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:58:47.313563 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:58:47.314036 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:58:47.314044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:58:47.314359 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 02:58:47.314592 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 02:58:47.315092 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":252,"threads_started":1}
I20260929 02:58:47.317700 31386 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34067
I20260929 02:58:47.318004 31386 heartbeater.cc:464] Registering TS with master...
I20260929 02:58:47.318728 31386 heartbeater.cc:511] Master 127.15.18.126:34067 requested a full tablet report, sending...
I20260929 02:58:47.320093 30929 ts_manager.cc:194] Registered new tserver with Master: 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835)
I20260929 02:58:47.320940 30929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:41683
I20260929 02:58:47.330690 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:58:47.348171 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.335109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.15.18.126:34067)
0929 02:58:47.335318 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.335324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.335588 (+ 264us) server_negotiation.cc:407] Connection header received
0929 02:58:47.335760 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.335764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.335833 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.335904 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:47.337276 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.338007 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.339308 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.339793 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.339922 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.342194 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:47.342199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:47.342203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:47.342239 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:47.345160 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:47.345964 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:47.345970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:47.345974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:47.346138 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:47.346597 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:47.346602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:47.346605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:47.347061 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:47.347317 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.347603 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.347830 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:58:47.352246 30929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:58:47.358954 30929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:58:47.378743 31119 heartbeater.cc:503] Master 127.15.18.126:34067 was elected leader, sending a full tablet report...
I20260929 02:58:47.382030 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.370666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:39415 (local address 127.0.0.1:33668)
0929 02:58:47.371981 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:47.372027 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:47.372187 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:47.373719 (+ 1532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:47.373727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:47.373777 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:47.374187 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.374200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.376180 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.376186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.378409 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.378421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.380923 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.380929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.381115 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.381173 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 02:58:47.381273 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:58:47.381644 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":2267,"threads_started":1}
I20260929 02:58:47.383455 31399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.370879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.15.18.66:39415)
0929 02:58:47.373119 (+ 2240us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.373127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.373153 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:58:47.373247 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.373254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.373500 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.374370 (+ 870us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:47.374577 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.376035 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.379273 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.380702 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.382724 (+ 2022us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.382945 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.383015 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.383181 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2090,"thread_start_us":1975,"threads_started":1}
I20260929 02:58:47.384163 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.371244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:33835 (local address 127.0.0.1:53312)
0929 02:58:47.373180 (+ 1936us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:47.373196 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:47.373378 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:47.375174 (+ 1796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:47.375179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:47.375203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:47.375616 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.375629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.378444 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.378449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.380477 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.380492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.383429 (+ 2937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.383436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.383630 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.383661 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:58:47.383735 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:58:47.383859 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1814,"thread_start_us":804,"threads_started":1}
I20260929 02:58:47.384415 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.371821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.15.18.67:33835)
0929 02:58:47.372795 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.372804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.373286 (+ 482us) server_negotiation.cc:407] Connection header received
0929 02:58:47.374535 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.374541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.374750 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.374903 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:47.375800 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.378234 (+ 2434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.380654 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.382961 (+ 2307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.383052 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.383239 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.383948 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.384106 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":743,"threads_started":1}
I20260929 02:58:47.388675 31321 tablet_service.cc:1467] Processing CreateTablet for tablet 54e238751fd042be9382d6b123071a1b (DEFAULT_TABLE table=test-table [id=42514af4a2e84d07ac784ad771e0501c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:47.390332 31321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e238751fd042be9382d6b123071a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:47.390641 31188 tablet_service.cc:1467] Processing CreateTablet for tablet 54e238751fd042be9382d6b123071a1b (DEFAULT_TABLE table=test-table [id=42514af4a2e84d07ac784ad771e0501c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:47.391198 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.373279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.0.0.1:44542)
0929 02:58:47.382247 (+ 8968us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:47.382264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:47.382355 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:47.385797 (+ 3442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:47.385801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:47.385827 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:47.386011 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.386022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.388149 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.388154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.389244 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.389254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.390476 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.390482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.390661 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.390695 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:58:47.390781 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 02:58:47.390926 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8864,"thread_start_us":12924,"threads_started":1}
I20260929 02:58:47.391269 31403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.373516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.15.18.65:41293)
0929 02:58:47.385221 (+ 11705us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.385230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.385257 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:58:47.385358 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.385366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.385612 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.386409 (+ 797us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:47.386672 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.388019 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.389409 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.390321 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.390441 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.390662 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.390835 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.390991 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11544,"thread_start_us":11385,"threads_started":1}
I20260929 02:58:47.392469 31188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e238751fd042be9382d6b123071a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:47.394080 31054 tablet_service.cc:1467] Processing CreateTablet for tablet 54e238751fd042be9382d6b123071a1b (DEFAULT_TABLE table=test-table [id=42514af4a2e84d07ac784ad771e0501c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:58:47.395103 31054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e238751fd042be9382d6b123071a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:58:47.404960 31404 tablet_bootstrap.cc:492] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: Bootstrap starting.
I20260929 02:58:47.406993 31405 tablet_bootstrap.cc:492] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: Bootstrap starting.
I20260929 02:58:47.408326 31406 tablet_bootstrap.cc:492] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: Bootstrap starting.
I20260929 02:58:47.409281 31404 tablet_bootstrap.cc:654] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:47.410392 31405 tablet_bootstrap.cc:654] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:47.410576 31404 log.cc:824] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:47.411458 31405 log.cc:824] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:47.411542 31406 tablet_bootstrap.cc:654] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:58:47.412703 31406 log.cc:824] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:58:47.414302 31404 tablet_bootstrap.cc:492] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: No bootstrap required, opened a new log
I20260929 02:58:47.414534 31404 ts_tablet_manager.cc:1397] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 02:58:47.415398 31406 tablet_bootstrap.cc:492] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: No bootstrap required, opened a new log
I20260929 02:58:47.415675 31405 tablet_bootstrap.cc:492] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: No bootstrap required, opened a new log
I20260929 02:58:47.415693 31406 ts_tablet_manager.cc:1397] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:58:47.415895 31405 ts_tablet_manager.cc:1397] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:58:47.421757 31404 raft_consensus.cc:348] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } } peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } }
I20260929 02:58:47.422097 31404 raft_consensus.cc:374] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:47.422219 31404 raft_consensus.cc:729] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a31d48ec1bc41dc8766d469412bfef3, State: Initialized, Role: FOLLOWER
I20260929 02:58:47.422763 31404 consensus_queue.cc:260] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } } peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } }
I20260929 02:58:47.425621 31406 raft_consensus.cc:348] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } } peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } }
I20260929 02:58:47.425617 31405 raft_consensus.cc:348] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } } peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } }
I20260929 02:58:47.426048 31405 raft_consensus.cc:374] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:47.426064 31406 raft_consensus.cc:374] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:58:47.426152 31406 raft_consensus.cc:729] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e72a441d1b45769ff4341eeb4be6a1, State: Initialized, Role: FOLLOWER
I20260929 02:58:47.426175 31405 raft_consensus.cc:729] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb465fabbbe54e20a59b2c7daba3e538, State: Initialized, Role: FOLLOWER
I20260929 02:58:47.426832 31406 consensus_queue.cc:260] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } } peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } }
I20260929 02:58:47.426919 31405 consensus_queue.cc:260] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } } peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } }
I20260929 02:58:47.428817 31386 heartbeater.cc:503] Master 127.15.18.126:34067 was elected leader, sending a full tablet report...
I20260929 02:58:47.429381 31404 ts_tablet_manager.cc:1428] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: Time spent starting tablet: real 0.015s user 0.006s sys 0.009s
I20260929 02:58:47.436884 31405 ts_tablet_manager.cc:1428] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: Time spent starting tablet: real 0.021s user 0.011s sys 0.009s
I20260929 02:58:47.436892 31406 ts_tablet_manager.cc:1428] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: Time spent starting tablet: real 0.021s user 0.011s sys 0.006s
I20260929 02:58:47.437207 31253 heartbeater.cc:503] Master 127.15.18.126:34067 was elected leader, sending a full tablet report...
I20260929 02:58:47.446919 31412 raft_consensus.cc:482] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:58:47.447211 31412 raft_consensus.cc:504] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } } peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } }
I20260929 02:58:47.448866 31412 leader_election.cc:288] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415), 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835)
I20260929 02:58:47.455233 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.449342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:59963 (local address 127.15.18.67:33835)
0929 02:58:47.449540 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.449546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.450173 (+ 627us) server_negotiation.cc:407] Connection header received
0929 02:58:47.450456 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.450461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.450536 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.450617 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:47.451151 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.451929 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.453596 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.454331 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.454499 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.454574 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.454876 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.454974 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:58:47.455327 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.449099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:33835 (local address 127.15.18.65:59963)
0929 02:58:47.450063 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:47.450083 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:47.450296 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:47.450668 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:47.450673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:47.450697 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:47.451020 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.451030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.452045 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.452049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.453461 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.453473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.454520 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.454527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.454711 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.454743 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:58:47.454850 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 02:58:47.454974 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":992,"threads_started":1}
I20260929 02:58:47.455447 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.449174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:39415 (local address 127.15.18.65:59387)
0929 02:58:47.450348 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0929 02:58:47.450371 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:58:47.450484 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:58:47.450863 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:58:47.450868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:58:47.450893 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:58:47.451227 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.451235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.452212 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.452216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.453598 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:58:47.453610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.454734 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.454740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:58:47.454892 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.454920 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:58:47.454992 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 02:58:47.455096 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":1,"thread_start_us":548,"threads_started":1}
I20260929 02:58:47.455458 31399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.449262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:59387 (local address 127.15.18.66:39415)
0929 02:58:47.449749 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.449755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.450446 (+ 691us) server_negotiation.cc:407] Connection header received
0929 02:58:47.450612 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.450618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.450688 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.450765 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:58:47.451359 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.452083 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.453736 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.454498 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.454649 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.454714 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.455031 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.455167 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260929 02:58:47.456314 31341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e238751fd042be9382d6b123071a1b" candidate_uuid: "cb465fabbbe54e20a59b2c7daba3e538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a31d48ec1bc41dc8766d469412bfef3" is_pre_election: true
I20260929 02:58:47.456563 31208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e238751fd042be9382d6b123071a1b" candidate_uuid: "cb465fabbbe54e20a59b2c7daba3e538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" is_pre_election: true
I20260929 02:58:47.456914 31341 raft_consensus.cc:2454] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb465fabbbe54e20a59b2c7daba3e538 in term 0.
I20260929 02:58:47.457046 31208 raft_consensus.cc:2454] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb465fabbbe54e20a59b2c7daba3e538 in term 0.
I20260929 02:58:47.457624 31010 leader_election.cc:302] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a31d48ec1bc41dc8766d469412bfef3, cb465fabbbe54e20a59b2c7daba3e538; no voters:
I20260929 02:58:47.458010 31412 raft_consensus.cc:2781] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:58:47.458141 31412 raft_consensus.cc:482] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:58:47.458241 31412 raft_consensus.cc:3037] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:47.460390 31412 raft_consensus.cc:504] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } } peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } }
I20260929 02:58:47.460928 31412 leader_election.cc:288] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 1 election: Requested vote from peers e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415), 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835)
I20260929 02:58:47.461380 31208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e238751fd042be9382d6b123071a1b" candidate_uuid: "cb465fabbbe54e20a59b2c7daba3e538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e72a441d1b45769ff4341eeb4be6a1"
I20260929 02:58:47.461390 31341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e238751fd042be9382d6b123071a1b" candidate_uuid: "cb465fabbbe54e20a59b2c7daba3e538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a31d48ec1bc41dc8766d469412bfef3"
I20260929 02:58:47.461548 31208 raft_consensus.cc:3037] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:47.461556 31341 raft_consensus.cc:3037] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:58:47.463642 31208 raft_consensus.cc:2454] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb465fabbbe54e20a59b2c7daba3e538 in term 1.
I20260929 02:58:47.463689 31341 raft_consensus.cc:2454] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb465fabbbe54e20a59b2c7daba3e538 in term 1.
I20260929 02:58:47.464032 31007 leader_election.cc:302] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb465fabbbe54e20a59b2c7daba3e538, e0e72a441d1b45769ff4341eeb4be6a1; no voters:
I20260929 02:58:47.464298 31412 raft_consensus.cc:2781] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:58:47.464723 31412 raft_consensus.cc:686] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 1 LEADER]: Becoming Leader. State: Replica: cb465fabbbe54e20a59b2c7daba3e538, State: Running, Role: LEADER
I20260929 02:58:47.465190 31412 consensus_queue.cc:237] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } } peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } }
I20260929 02:58:47.469522 30929 catalog_manager.cc:5495] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb465fabbbe54e20a59b2c7daba3e538 (127.15.18.65). New cstate: current_term: 1 leader_uuid: "cb465fabbbe54e20a59b2c7daba3e538" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } health_report { overall_health: UNKNOWN } } }
I20260929 02:58:47.505956 31403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.500795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.15.18.65:41293)
0929 02:58:47.500958 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.500962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.501143 (+ 181us) server_negotiation.cc:407] Connection header received
0929 02:58:47.501315 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.501319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.501397 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.501515 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:58:47.501859 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.502564 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.504032 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.504428 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.504513 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.504990 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:58:47.505466 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:58:47.505566 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.505674 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.505738 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:58:47.513828 31208 raft_consensus.cc:1261] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1 [term 1 FOLLOWER]: Refusing update from remote peer cb465fabbbe54e20a59b2c7daba3e538: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:58:47.513828 31341 raft_consensus.cc:1261] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3 [term 1 FOLLOWER]: Refusing update from remote peer cb465fabbbe54e20a59b2c7daba3e538: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:58:47.514865 31412 consensus_queue.cc:1035] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:58:47.515173 31415 consensus_queue.cc:1035] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:58:47.525931 31418 mvcc.cc:205] Tried to move back new op lower bound from 7334505379887222784 to 7334505379701129216. Current Snapshot: MvccSnapshot[applied={T|T < 7334505379887222784}]
I20260929 02:58:47.527155 31420 mvcc.cc:205] Tried to move back new op lower bound from 7334505379887222784 to 7334505379701129216. Current Snapshot: MvccSnapshot[applied={T|T < 7334505379887222784}]
I20260929 02:58:47.541014 30929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:58:47.541241 30929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:58:47.544730 30929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Sending DeleteTablet for 3 replicas of tablet 54e238751fd042be9382d6b123071a1b
I20260929 02:58:47.545961 31188 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e238751fd042be9382d6b123071a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:47 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:33668
I20260929 02:58:47.545926 31321 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e238751fd042be9382d6b123071a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:47 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:53312
I20260929 02:58:47.546835 31428 tablet_replica.cc:330] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: stopping tablet replica
I20260929 02:58:47.546947 31054 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e238751fd042be9382d6b123071a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:58:47 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:44542
I20260929 02:58:47.548051 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 30991
I20260929 02:58:47.548470 31428 raft_consensus.cc:2229] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:58:47.549005 31428 raft_consensus.cc:2258] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:47.551379 31428 ts_tablet_manager.cc:1905] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:47.558301 31428 ts_tablet_manager.cc:1918] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:47.558527 31428 log.cc:1196] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/54e238751fd042be9382d6b123071a1b
I20260929 02:58:47.559108 31428 ts_tablet_manager.cc:1939] T 54e238751fd042be9382d6b123071a1b P e0e72a441d1b45769ff4341eeb4be6a1: Deleting consensus metadata
I20260929 02:58:47.561046 30914 catalog_manager.cc:4841] TS e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415): tablet 54e238751fd042be9382d6b123071a1b (table test-table [id=42514af4a2e84d07ac784ad771e0501c]) successfully deleted
I20260929 02:58:47.561209 31429 tablet_replica.cc:330] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: stopping tablet replica
I20260929 02:58:47.561654 31429 raft_consensus.cc:2229] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:58:47.562152 31429 raft_consensus.cc:2258] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:47.564123 31429 ts_tablet_manager.cc:1905] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:47.564837 31430 tablet_replica.cc:330] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: stopping tablet replica
I20260929 02:58:47.565315 31430 raft_consensus.cc:2229] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:58:47.566033 31430 raft_consensus.cc:2258] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:58:47.568100 31430 ts_tablet_manager.cc:1905] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:58:47.570940 31429 ts_tablet_manager.cc:1918] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:47.571136 31429 log.cc:1196] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/54e238751fd042be9382d6b123071a1b
I20260929 02:58:47.572024 31429 ts_tablet_manager.cc:1939] T 54e238751fd042be9382d6b123071a1b P 1a31d48ec1bc41dc8766d469412bfef3: Deleting consensus metadata
I20260929 02:58:47.573856 30917 catalog_manager.cc:4841] TS 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835): tablet 54e238751fd042be9382d6b123071a1b (table test-table [id=42514af4a2e84d07ac784ad771e0501c]) successfully deleted
Sep 29 02:58:47 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:58:47.575217 31430 ts_tablet_manager.cc:1918] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:58:47.575405 31430 log.cc:1196] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/54e238751fd042be9382d6b123071a1b
I20260929 02:58:47.575964 31430 ts_tablet_manager.cc:1939] T 54e238751fd042be9382d6b123071a1b P cb465fabbbe54e20a59b2c7daba3e538: Deleting consensus metadata
I20260929 02:58:47.577677 30914 catalog_manager.cc:4841] TS cb465fabbbe54e20a59b2c7daba3e538 (127.15.18.65:41293): tablet 54e238751fd042be9382d6b123071a1b (table test-table [id=42514af4a2e84d07ac784ad771e0501c]) successfully deleted
I20260929 02:58:47.581832 31403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.549410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.15.18.65:41293)
0929 02:58:47.549597 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.549603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.564175 (+ 14572us) server_negotiation.cc:407] Connection header received
0929 02:58:47.564388 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.564394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.564500 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.564607 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:47.566871 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.567990 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.569952 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.570677 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.570772 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.575756 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:47.575767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:47.575780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:47.575818 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:47.578779 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:47.579674 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:47.579681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:47.579684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:47.579859 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:47.580306 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:47.580309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:47.580311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:47.580639 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:47.580884 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.581243 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.581411 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:58:47.582505 31114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:47.678793 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 30991
I20260929 02:58:47.693230 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 31125
Sep 29 02:58:47 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:58:47.707823 31399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.694038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.15.18.66:39415)
0929 02:58:47.694193 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.694198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.694223 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:58:47.694303 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.694306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.694367 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.694452 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:47.695662 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.696431 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.697695 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.698085 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.698172 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.702355 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:47.702371 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:47.702388 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:47.702424 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:47.705255 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:47.705841 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:47.705848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:47.705851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:47.705978 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:47.706309 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:47.706313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:47.706315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:47.706749 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:47.706978 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.707230 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.707415 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 02:58:47.708482 31248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:47.796268 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 31125
I20260929 02:58:47.809989 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 31258
Sep 29 02:58:47 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:58:47.824738 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:58:47.810809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.15.18.67:33835)
0929 02:58:47.810959 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 02:58:47.810963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:58:47.810981 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:58:47.811132 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:58:47.811136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:58:47.811206 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:58:47.811296 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:58:47.812658 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.813520 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.814802 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:58:47.815172 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:58:47.815257 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:58:47.819670 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:58:47.819686 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:58:47.819701 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:58:47.819736 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:58:47.822234 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:47.822839 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:47.822845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:47.822847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:47.822977 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:58:47.823250 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:58:47.823252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:58:47.823254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:58:47.823618 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:58:47.823834 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 02:58:47.824177 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 02:58:47.824369 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:58:47.825380 31381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:47.912643 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 31258
I20260929 02:58:47.925633 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 30898
I20260929 02:58:47.926712 30960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:58:48.034189 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 30898
I20260929 02:59:08.051451 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:34067
--webserver_interface=127.15.18.126
--webserver_port=39987
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:59:08.369522 31436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:08.369812 31436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:08.369944 31436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:08.377286 31436 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:59:08.377408 31436 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:59:08.377472 31436 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:59:08.377552 31436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:59:08.377645 31436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:08.377753 31436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:08.377822 31436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:59:08.377904 31436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:59:08.387792 31436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:34067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=39987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:08.389067 31436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:08.390584 31436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:08.411732 31442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:08.411705 31443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:08.411638 31445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:08.412386 31436 server_base.cc:1034] running on GCE node
I20260929 02:59:08.413267 31436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:08.414847 31436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:08.416134 31436 hybrid_clock.cc:648] HybridClock initialized: now 1790650748416067 us; error 30 us; skew 500 ppm
I20260929 02:59:08.416527 31436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:08 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650748, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:08.424347 31436 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:59:08.425908 31436 webserver.cc:460] Webserver started at http://127.15.18.126:39987/ using document root <none> and password file <none>
I20260929 02:59:08.426651 31436 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:08.426792 31436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:08.432890 31436 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:59:08.435796 31452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:08.437105 31436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:59:08.437280 31436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "9997807127644aa6b091762cc2d0f82a"
format_stamp: "Formatted at 2026-09-29 02:58:45 on dist-test-slave-q5f0"
server_key: "099f5b19c7b3226d6fa5f4932b049fe7"
server_key_iv: "d191dbe36babe2fc1bc49306313afc79"
server_key_version: "encryptionkey@0"
I20260929 02:59:08.438313 31436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:08.450039 31436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:08.454298 31436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:08.454555 31436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:08.471683 31436 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:34067
I20260929 02:59:08.471690 31503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:34067 every 8 connection(s)
I20260929 02:59:08.473165 31436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:59:08.480026 31504 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:59:08.483249 31504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Bootstrap starting.
I20260929 02:59:08.483577 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 31436
I20260929 02:59:08.484390 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:41293
--local_ip_for_outbound_sockets=127.15.18.65
--tserver_master_addrs=127.15.18.126:34067
--webserver_port=46869
--webserver_interface=127.15.18.65
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:59:08.502686 31504 log.cc:824] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:08.514663 31504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:59:08.515154 31504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Bootstrap complete.
I20260929 02:59:08.523944 31504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:59:08.524773 31504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9997807127644aa6b091762cc2d0f82a, State: Initialized, Role: FOLLOWER
I20260929 02:59:08.525343 31504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:59:08.525565 31504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:59:08.525672 31504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:59:08.525830 31504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:59:08.529608 31504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:59:08.530104 31504 leader_election.cc:302] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9997807127644aa6b091762cc2d0f82a; no voters:
I20260929 02:59:08.530668 31504 leader_election.cc:288] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:59:08.530803 31508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:59:08.532059 31508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 2 LEADER]: Becoming Leader. State: Replica: 9997807127644aa6b091762cc2d0f82a, State: Running, Role: LEADER
I20260929 02:59:08.532639 31508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:59:08.532966 31504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:59:08.535140 31510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9997807127644aa6b091762cc2d0f82a. Latest consensus state: current_term: 2 leader_uuid: "9997807127644aa6b091762cc2d0f82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } } }
I20260929 02:59:08.535609 31510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:08.536166 31509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9997807127644aa6b091762cc2d0f82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } } }
I20260929 02:59:08.536487 31509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:08.544747 31515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:59:08.553397 31515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:59:08.554916 31515 catalog_manager.cc:1259] Loaded cluster ID: 039f016e23fc484fa18a28ad0ff39b97
I20260929 02:59:08.555084 31515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:59:08.559360 31515 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:59:08.561039 31515 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Loaded TSK: 0
I20260929 02:59:08.597246 31515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Generated new TSK 1
I20260929 02:59:08.597723 31515 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:59:08.839639 31506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:08.839915 31506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:08.840032 31506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:08.847230 31506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:08.847338 31506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:08.847479 31506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:59:08.857669 31506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:41293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=46869
--tserver_master_addrs=127.15.18.126:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:08.858934 31506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:08.860283 31506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:08.874850 31532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:08.874832 31531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:08.875208 31534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:08.875579 31506 server_base.cc:1034] running on GCE node
I20260929 02:59:08.876466 31506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:08.877751 31506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:08.879019 31506 hybrid_clock.cc:648] HybridClock initialized: now 1790650748878953 us; error 50 us; skew 500 ppm
I20260929 02:59:08.879431 31506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:08 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650748, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:08.883533 31506 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:59:08.885309 31506 webserver.cc:460] Webserver started at http://127.15.18.65:46869/ using document root <none> and password file <none>
I20260929 02:59:08.886200 31506 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:08.886363 31506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:08.893481 31506 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 02:59:08.896845 31541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:08.898511 31506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:59:08.898685 31506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "cb465fabbbe54e20a59b2c7daba3e538"
format_stamp: "Formatted at 2026-09-29 02:58:46 on dist-test-slave-q5f0"
server_key: "9655e6851ee33470683e906ca5ac65d5"
server_key_iv: "6636ac715242fab654e3f03cd9d0e850"
server_key_version: "encryptionkey@0"
I20260929 02:59:08.900060 31506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:08.926731 31506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:08.931730 31506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:08.932018 31506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:08.933148 31506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:08.934721 31506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:08.934824 31506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:08.934927 31506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:08.935005 31506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:08.969901 31506 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:41293
I20260929 02:59:08.969956 31653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:41293 every 8 connection(s)
I20260929 02:59:08.971778 31506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:59:08.977030 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 31506
I20260929 02:59:08.977733 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:39415
--local_ip_for_outbound_sockets=127.15.18.66
--tserver_master_addrs=127.15.18.126:34067
--webserver_port=33499
--webserver_interface=127.15.18.66
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:59:08 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650748, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:08.997584 31657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:08.974931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:38939)
0929 02:59:08.977750 (+ 2819us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:08.977804 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:08.979905 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:08.981305 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:08.981316 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:08.981944 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:08.982491 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:08.982508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:08.984237 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:08.984241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:08.985603 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:08.985618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:08.986732 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:08.986739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:08.987007 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:08.987809 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:08.987836 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:08.989973 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:08.993224 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:08.993234 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:08.993250 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:08.993670 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:08.994080 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:08.994085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:08.994089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:08.994323 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:08.995062 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:08.995070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:08.995408 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 02:59:08.995704 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 02:59:08.996265 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2527,"thread_start_us":2325,"threads_started":1}
I20260929 02:59:08.997584 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:08.974755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:38939 (local address 127.15.18.126:34067)
0929 02:59:08.975465 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:08.975484 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:08.978217 (+ 2733us) server_negotiation.cc:407] Connection header received
0929 02:59:08.980427 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:08.980441 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:08.980787 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:08.981131 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:08.982717 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:08.984072 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:08.985815 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:08.986492 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:08.986637 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:08.990247 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:08.990255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:08.990269 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:08.990310 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:08.993078 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:08.993802 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:08.993810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:08.993815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:08.993971 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:08.994419 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:08.994424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:08.994428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:08.994928 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:08.995105 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:08.995965 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 02:59:08.996371 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":292,"threads_started":1}
I20260929 02:59:09.000706 31654 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34067
I20260929 02:59:09.001017 31654 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:09.001768 31654 heartbeater.cc:511] Master 127.15.18.126:34067 requested a full tablet report, sending...
I20260929 02:59:09.003895 31469 ts_manager.cc:194] Registered new tserver with Master: cb465fabbbe54e20a59b2c7daba3e538 (127.15.18.65:41293)
I20260929 02:59:09.005405 31469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:38939
W20260929 02:59:09.298657 31658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:09.298962 31658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:09.299094 31658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:09.306406 31658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:09.306527 31658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:09.306681 31658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:59:09.316409 31658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:39415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=33499
--tserver_master_addrs=127.15.18.126:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:09.317700 31658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:09.318982 31658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:09.331725 31664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:09.331804 31666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:09.332139 31663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:09.332504 31658 server_base.cc:1034] running on GCE node
I20260929 02:59:09.333344 31658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:09.334321 31658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:09.335577 31658 hybrid_clock.cc:648] HybridClock initialized: now 1790650749335491 us; error 36 us; skew 500 ppm
I20260929 02:59:09.335927 31658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:09 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:09.338932 31658 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:59:09.340410 31658 webserver.cc:460] Webserver started at http://127.15.18.66:33499/ using document root <none> and password file <none>
I20260929 02:59:09.341061 31658 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:09.341178 31658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:09.346408 31658 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:59:09.349093 31673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:09.350389 31658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:59:09.350533 31658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "e0e72a441d1b45769ff4341eeb4be6a1"
format_stamp: "Formatted at 2026-09-29 02:58:46 on dist-test-slave-q5f0"
server_key: "6358f5e6bf70708f01b81f1f8515bacc"
server_key_iv: "f6dc3eef243486c90a808453a4bc0bca"
server_key_version: "encryptionkey@0"
I20260929 02:59:09.351449 31658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:09.368455 31658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:09.372583 31658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:09.372838 31658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:09.373790 31658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:09.375113 31658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:09.375198 31658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:09.375293 31658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:09.375357 31658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:09.404901 31658 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:39415
I20260929 02:59:09.404956 31785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:39415 every 8 connection(s)
I20260929 02:59:09.406309 31658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:59:09.411692 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 31658
I20260929 02:59:09.412405 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:33835
--local_ip_for_outbound_sockets=127.15.18.67
--tserver_master_addrs=127.15.18.126:34067
--webserver_port=33767
--webserver_interface=127.15.18.67
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:59:09 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:09.427604 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.408751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:53861 (local address 127.15.18.126:34067)
0929 02:59:09.408990 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:09.408997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:09.410135 (+ 1138us) server_negotiation.cc:407] Connection header received
0929 02:59:09.411961 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:09.411968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:09.412059 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:09.412173 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:09.413775 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.414732 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.416319 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.416916 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.417032 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.420540 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:09.420546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:09.420550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:09.420590 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:09.423616 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:09.424587 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:09.424593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:09.424597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:09.424755 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:09.425292 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:09.425297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:09.425300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:09.425668 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:09.425814 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:09.426772 (+ 958us) server_negotiation.cc:299] Negotiation successful
0929 02:59:09.427145 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:59:09.428629 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.409227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:53861)
0929 02:59:09.409978 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:09.410015 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:09.411696 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:09.412418 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:09.412426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:09.413021 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:09.413586 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.413604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.414891 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.414895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.416189 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.416201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.417102 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.417107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.417315 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.417943 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:09.417965 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:09.420292 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:09.423808 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:09.423819 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:09.423838 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:09.424440 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:09.424875 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:09.424881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:09.424885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:09.425164 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:09.425810 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:09.425849 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:09.426325 (+ 476us) client_negotiation.cc:769] Sending connection context
0929 02:59:09.426629 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 02:59:09.427207 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":277,"threads_started":1}
I20260929 02:59:09.430598 31786 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34067
I20260929 02:59:09.430984 31786 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:09.431916 31786 heartbeater.cc:511] Master 127.15.18.126:34067 requested a full tablet report, sending...
I20260929 02:59:09.433586 31469 ts_manager.cc:194] Registered new tserver with Master: e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415)
I20260929 02:59:09.434419 31469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:53861
W20260929 02:59:09.734872 31789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:09.735174 31789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:09.735330 31789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:09.742659 31789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:09.742790 31789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:09.742957 31789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:59:09.752748 31789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:33835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=33767
--tserver_master_addrs=127.15.18.126:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:09.754127 31789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:09.755560 31789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:09.768278 31794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:09.768321 31797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:09.768348 31795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:09.769342 31789 server_base.cc:1034] running on GCE node
I20260929 02:59:09.769945 31789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:09.770823 31789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:09.772013 31789 hybrid_clock.cc:648] HybridClock initialized: now 1790650749771972 us; error 45 us; skew 500 ppm
I20260929 02:59:09.772300 31789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:09 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:09.775378 31789 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:59:09.776775 31789 webserver.cc:460] Webserver started at http://127.15.18.67:33767/ using document root <none> and password file <none>
I20260929 02:59:09.777392 31789 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:09.777534 31789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:09.782641 31789 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 02:59:09.785271 31804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:09.786660 31789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:59:09.786803 31789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "1a31d48ec1bc41dc8766d469412bfef3"
format_stamp: "Formatted at 2026-09-29 02:58:47 on dist-test-slave-q5f0"
server_key: "801833e89d12ae83d5f8212019f8bcfc"
server_key_iv: "0359150d0df66a7337616fe8d0465669"
server_key_version: "encryptionkey@0"
I20260929 02:59:09.787814 31789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:09.819195 31789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:09.823335 31789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:09.823635 31789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:09.824587 31789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:09.825915 31789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:09.826012 31789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:09.826114 31789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:09.826184 31789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:09.855239 31789 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:33835
I20260929 02:59:09.855307 31916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:33835 every 8 connection(s)
I20260929 02:59:09.856683 31789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:59:09.859135 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 31789
Sep 29 02:59:09 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650749, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:09.878539 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.860015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.15.18.126:34067)
0929 02:59:09.860474 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:09.860480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:09.863823 (+ 3343us) server_negotiation.cc:407] Connection header received
0929 02:59:09.863976 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:09.863980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:09.864035 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:09.864118 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:09.865976 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.867079 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.868738 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.869398 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.869531 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.872424 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:09.872429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:09.872432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:09.872463 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:09.875391 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:09.876261 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:09.876268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:09.876271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:09.876432 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:09.876854 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:09.876858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:09.876860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:09.877202 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:09.877446 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:09.877858 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 02:59:09.878112 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"spinlock_wait_cycles":12544,"thread_start_us":234,"threads_started":1}
I20260929 02:59:09.878636 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.859348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:51049 (local address 127.15.18.126:34067)
0929 02:59:09.859588 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:09.859595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:09.861160 (+ 1565us) server_negotiation.cc:407] Connection header received
0929 02:59:09.862958 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:09.862964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:09.863056 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:09.863182 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:09.864777 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.865867 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.867928 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.868568 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.868771 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.872770 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:09.872774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:09.872777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:09.872819 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:09.875570 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:09.876278 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:09.876282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:09.876284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:09.876434 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:09.876851 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:09.876854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:09.876856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:09.877168 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:09.877293 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:09.877978 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 02:59:09.878271 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":79}
I20260929 02:59:09.880124 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.859924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:51049)
0929 02:59:09.860995 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:09.861039 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:09.862715 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:09.863411 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:09.863420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:09.864008 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:09.864600 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.864619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.866040 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.866046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.867787 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.867799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.868846 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.868855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.869152 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.870040 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:09.870063 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:09.872559 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:09.875712 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:09.875722 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:09.875739 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:09.876157 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:09.876536 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:09.876541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:09.876545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:09.876739 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:09.877285 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:09.877294 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:09.877628 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 02:59:09.877868 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 02:59:09.878595 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":635,"threads_started":1}
I20260929 02:59:09.881496 31917 heartbeater.cc:347] Connected to a master server at 127.15.18.126:34067
I20260929 02:59:09.881825 31917 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:09.882575 31917 heartbeater.cc:511] Master 127.15.18.126:34067 requested a full tablet report, sending...
I20260929 02:59:09.884125 31469 ts_manager.cc:194] Registered new tserver with Master: 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835)
I20260929 02:59:09.884943 31469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:51049
I20260929 02:59:09.893784 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:59:09.900017 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.894769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.15.18.126:34067)
0929 02:59:09.894923 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:09.894928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:09.895322 (+ 394us) server_negotiation.cc:407] Connection header received
0929 02:59:09.895530 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:09.895535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:09.895678 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:09.895787 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:59:09.896146 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.896891 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.898352 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.898748 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.898843 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.899274 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:59:09.899486 (+ 212us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 02:59:09.899832 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55438: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 02:59:09.900122 31656 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55438: token expired
W20260929 02:59:09.900665 31392 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.18.126:34067)
I20260929 02:59:09.913167 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.901235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.15.18.126:34067)
0929 02:59:09.901412 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:09.901416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:09.901428 (+ 12us) server_negotiation.cc:407] Connection header received
0929 02:59:09.901473 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:09.901476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:09.901526 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:09.901598 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:09.902985 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.903707 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.905162 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.905588 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.905704 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.908418 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:09.908423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:09.908426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:09.908455 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:09.910760 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:09.911362 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:09.911367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:09.911370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:09.911484 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:09.911836 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:09.911841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:09.911845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:09.912153 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:09.912392 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:09.912662 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 02:59:09.912867 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:59:09.916620 31469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:59:09.923621 31469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:59:09.946260 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.936598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:39415 (local address 127.0.0.1:37432)
0929 02:59:09.938072 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:09.938089 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:09.938205 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:09.939095 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:09.939100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:09.939129 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:09.939586 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.939599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.941777 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.941783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.943899 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.943911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.945247 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.945253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.945433 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.945482 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 02:59:09.945603 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 02:59:09.945800 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"mutex_wait_us":114,"thread_start_us":1055,"threads_started":1}
I20260929 02:59:09.946451 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.935070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:33835 (local address 127.0.0.1:46314)
0929 02:59:09.937488 (+ 2418us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:09.937505 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:09.937621 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:09.938623 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:09.938630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:09.938669 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:09.939180 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.939189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.941229 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.941234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.943890 (+ 2656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.943900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.945657 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.945663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.945836 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.945865 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:59:09.945962 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 02:59:09.946078 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2299,"mutex_wait_us":224,"spinlock_wait_cycles":1280,"thread_start_us":1713,"threads_started":1}
I20260929 02:59:09.946905 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.935214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.15.18.67:33835)
0929 02:59:09.936184 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:09.936189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:09.937815 (+ 1626us) server_negotiation.cc:407] Connection header received
0929 02:59:09.937906 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:09.937913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:09.938168 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:09.938351 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:09.939424 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.941098 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.944054 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.945445 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.945576 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.945794 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:09.946010 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 02:59:09.946217 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":683,"threads_started":1}
I20260929 02:59:09.946898 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.934958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.15.18.65:41293)
0929 02:59:09.935662 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:09.935672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:09.937413 (+ 1741us) server_negotiation.cc:407] Connection header received
0929 02:59:09.937910 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:09.937917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:09.938168 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:09.938360 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:09.939409 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.940880 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.944054 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.945478 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.945576 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.945804 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:09.946426 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 02:59:09.946586 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":395,"threads_started":1}
I20260929 02:59:09.947171 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.936808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.15.18.66:39415)
0929 02:59:09.938670 (+ 1862us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:09.938676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:09.938697 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:59:09.938773 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:09.938778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:09.938928 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:09.939091 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:09.939761 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.941656 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.944084 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.945047 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.945198 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.945397 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:09.946731 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0929 02:59:09.946874 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1722,"thread_start_us":1596,"threads_started":1}
I20260929 02:59:09.947674 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.934737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.0.0.1:57892)
0929 02:59:09.937221 (+ 2484us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:09.937281 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:09.937578 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:09.938630 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:09.938634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:09.938677 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:09.939170 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.939183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.941050 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.941055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.943871 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.943887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.945997 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.946003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.946191 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:09.946228 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:59:09.947146 (+ 918us) client_negotiation.cc:240] Negotiation successful
0929 02:59:09.947287 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2186,"mutex_wait_us":196,"spinlock_wait_cycles":4352,"thread_start_us":1047,"threads_started":1}
I20260929 02:59:09.950388 31852 tablet_service.cc:1467] Processing CreateTablet for tablet ac72c9fba7fe438d8fad419d2063292a (DEFAULT_TABLE table=test-table [id=88704e2eb8784275843916231202c5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:09.950587 31721 tablet_service.cc:1467] Processing CreateTablet for tablet ac72c9fba7fe438d8fad419d2063292a (DEFAULT_TABLE table=test-table [id=88704e2eb8784275843916231202c5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:09.951606 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac72c9fba7fe438d8fad419d2063292a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:09.951800 31721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac72c9fba7fe438d8fad419d2063292a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:09.953325 31589 tablet_service.cc:1467] Processing CreateTablet for tablet ac72c9fba7fe438d8fad419d2063292a (DEFAULT_TABLE table=test-table [id=88704e2eb8784275843916231202c5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:09.954535 31589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac72c9fba7fe438d8fad419d2063292a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:09.962621 31933 tablet_bootstrap.cc:492] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: Bootstrap starting.
I20260929 02:59:09.966176 31933 tablet_bootstrap.cc:654] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:09.967135 31933 log.cc:824] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:09.969506 31936 tablet_bootstrap.cc:492] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Bootstrap starting.
I20260929 02:59:09.969812 31933 tablet_bootstrap.cc:492] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: No bootstrap required, opened a new log
I20260929 02:59:09.970064 31933 ts_tablet_manager.cc:1397] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260929 02:59:09.971736 31935 tablet_bootstrap.cc:492] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: Bootstrap starting.
I20260929 02:59:09.973021 31936 tablet_bootstrap.cc:654] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:09.974385 31936 log.cc:824] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:09.975246 31935 tablet_bootstrap.cc:654] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:09.976564 31935 log.cc:824] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:09.980844 31933 raft_consensus.cc:348] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:09.981298 31933 raft_consensus.cc:374] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:09.981393 31933 raft_consensus.cc:729] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a31d48ec1bc41dc8766d469412bfef3, State: Initialized, Role: FOLLOWER
I20260929 02:59:09.982102 31933 consensus_queue.cc:260] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:09.984026 31936 tablet_bootstrap.cc:492] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: No bootstrap required, opened a new log
I20260929 02:59:09.984357 31936 ts_tablet_manager.cc:1397] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260929 02:59:09.992545 31939 raft_consensus.cc:482] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:59:09.992825 31939 raft_consensus.cc:504] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:09.994151 31933 ts_tablet_manager.cc:1428] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: Time spent starting tablet: real 0.024s user 0.011s sys 0.006s
I20260929 02:59:09.994495 31935 tablet_bootstrap.cc:492] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: No bootstrap required, opened a new log
I20260929 02:59:09.994788 31935 ts_tablet_manager.cc:1397] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.007s
I20260929 02:59:09.994824 31917 heartbeater.cc:503] Master 127.15.18.126:34067 was elected leader, sending a full tablet report...
I20260929 02:59:10.001493 31936 raft_consensus.cc:348] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:10.001977 31936 raft_consensus.cc:374] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:10.002076 31936 raft_consensus.cc:729] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb465fabbbe54e20a59b2c7daba3e538, State: Initialized, Role: FOLLOWER
I20260929 02:59:10.002775 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.995099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.15.18.67:49577)
0929 02:59:09.995577 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:09.995596 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:09.995726 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:09.996243 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:09.996247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:09.996272 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:09.996447 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.996458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.997875 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:09.997880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:09.999925 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:09.999943 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.001924 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.001931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:10.002164 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:10.002201 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:59:10.002294 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 02:59:10.002446 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326}
I20260929 02:59:10.002856 31939 leader_election.cc:288] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb465fabbbe54e20a59b2c7daba3e538 (127.15.18.65:41293), e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415)
I20260929 02:59:10.002789 31936 consensus_queue.cc:260] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:10.003645 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.995199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:49577 (local address 127.15.18.65:41293)
0929 02:59:09.995405 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:09.995409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:09.995839 (+ 430us) server_negotiation.cc:407] Connection header received
0929 02:59:09.995916 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:09.995921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:09.995994 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:09.996073 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:09.996624 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:09.997708 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.000128 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.001573 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.001749 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:10.001830 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:10.003182 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0929 02:59:10.003340 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:59:10.007639 31935 raft_consensus.cc:348] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:10.008185 31935 raft_consensus.cc:374] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:10.008535 31654 heartbeater.cc:503] Master 127.15.18.126:34067 was elected leader, sending a full tablet report...
I20260929 02:59:10.008862 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac72c9fba7fe438d8fad419d2063292a" candidate_uuid: "1a31d48ec1bc41dc8766d469412bfef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb465fabbbe54e20a59b2c7daba3e538" is_pre_election: true
I20260929 02:59:10.009332 31609 raft_consensus.cc:2454] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a31d48ec1bc41dc8766d469412bfef3 in term 0.
I20260929 02:59:10.009984 31935 raft_consensus.cc:729] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e72a441d1b45769ff4341eeb4be6a1, State: Initialized, Role: FOLLOWER
I20260929 02:59:10.010061 31805 leader_election.cc:302] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a31d48ec1bc41dc8766d469412bfef3, cb465fabbbe54e20a59b2c7daba3e538; no voters:
I20260929 02:59:10.010694 31939 raft_consensus.cc:2781] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:59:10.010679 31935 consensus_queue.cc:260] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:10.010890 31939 raft_consensus.cc:482] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:59:10.011096 31939 raft_consensus.cc:3037] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:10.012141 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.996389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:48761 (local address 127.15.18.66:39415)
0929 02:59:09.996760 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:09.996768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:10.003092 (+ 6324us) server_negotiation.cc:407] Connection header received
0929 02:59:10.004141 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:10.004147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:10.004235 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:10.004301 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:10.004954 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.006152 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.009745 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.010952 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.011077 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:10.011151 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:10.011769 (+ 618us) server_negotiation.cc:299] Negotiation successful
0929 02:59:10.011898 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260929 02:59:10.012812 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:09.995933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:39415 (local address 127.15.18.67:48761)
0929 02:59:10.002995 (+ 7062us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:10.003015 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:10.003960 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:10.004504 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:10.004508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:10.004534 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:10.004746 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:10.004756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.006389 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.006395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:10.008272 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:10.008286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.011321 (+ 3035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.011327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:10.011557 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:10.011590 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:59:10.012351 (+ 761us) client_negotiation.cc:240] Negotiation successful
0929 02:59:10.012502 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6957,"thread_start_us":11920,"threads_started":1}
I20260929 02:59:10.014042 31786 heartbeater.cc:503] Master 127.15.18.126:34067 was elected leader, sending a full tablet report...
I20260929 02:59:10.014360 31936 ts_tablet_manager.cc:1428] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Time spent starting tablet: real 0.030s user 0.015s sys 0.006s
I20260929 02:59:10.014607 31935 ts_tablet_manager.cc:1428] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: Time spent starting tablet: real 0.020s user 0.017s sys 0.000s
I20260929 02:59:10.015861 31939 raft_consensus.cc:504] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:10.015878 31741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac72c9fba7fe438d8fad419d2063292a" candidate_uuid: "1a31d48ec1bc41dc8766d469412bfef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" is_pre_election: true
I20260929 02:59:10.016374 31741 raft_consensus.cc:2454] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a31d48ec1bc41dc8766d469412bfef3 in term 0.
I20260929 02:59:10.016624 31939 leader_election.cc:288] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [CANDIDATE]: Term 1 election: Requested vote from peers cb465fabbbe54e20a59b2c7daba3e538 (127.15.18.65:41293), e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415)
I20260929 02:59:10.017007 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac72c9fba7fe438d8fad419d2063292a" candidate_uuid: "1a31d48ec1bc41dc8766d469412bfef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb465fabbbe54e20a59b2c7daba3e538"
I20260929 02:59:10.017213 31609 raft_consensus.cc:3037] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:10.017765 31741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac72c9fba7fe438d8fad419d2063292a" candidate_uuid: "1a31d48ec1bc41dc8766d469412bfef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e72a441d1b45769ff4341eeb4be6a1"
I20260929 02:59:10.017942 31741 raft_consensus.cc:3037] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:10.020043 31609 raft_consensus.cc:2454] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a31d48ec1bc41dc8766d469412bfef3 in term 1.
I20260929 02:59:10.020406 31805 leader_election.cc:302] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a31d48ec1bc41dc8766d469412bfef3, cb465fabbbe54e20a59b2c7daba3e538; no voters:
I20260929 02:59:10.020690 31939 raft_consensus.cc:2781] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:59:10.020851 31939 raft_consensus.cc:686] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 1 LEADER]: Becoming Leader. State: Replica: 1a31d48ec1bc41dc8766d469412bfef3, State: Running, Role: LEADER
I20260929 02:59:10.020918 31741 raft_consensus.cc:2454] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a31d48ec1bc41dc8766d469412bfef3 in term 1.
I20260929 02:59:10.021346 31939 consensus_queue.cc:237] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:10.025771 31468 catalog_manager.cc:5495] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67). New cstate: current_term: 1 leader_uuid: "1a31d48ec1bc41dc8766d469412bfef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } health_report { overall_health: UNKNOWN } } }
I20260929 02:59:10.045764 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.040577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.15.18.67:33835)
0929 02:59:10.040775 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:10.040781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:10.040801 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:59:10.040967 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:10.040971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:10.041093 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:10.041225 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:59:10.041595 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.042399 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.043771 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.044154 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.044268 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:10.044737 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:59:10.045224 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:59:10.045353 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:10.045438 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 02:59:10.045518 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:59:10.053601 31741 raft_consensus.cc:1261] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1 [term 1 FOLLOWER]: Refusing update from remote peer 1a31d48ec1bc41dc8766d469412bfef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:59:10.053673 31609 raft_consensus.cc:1261] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Refusing update from remote peer 1a31d48ec1bc41dc8766d469412bfef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:59:10.054549 31940 consensus_queue.cc:1035] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:59:10.054997 31939 consensus_queue.cc:1035] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:59:10.065726 31946 mvcc.cc:205] Tried to move back new op lower bound from 7334505472210419712 to 7334505472091430912. Current Snapshot: MvccSnapshot[applied={T|T < 7334505472210419712}]
I20260929 02:59:10.078853 31468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:59:10.079134 31468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:59:10.082545 31468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Sending DeleteTablet for 3 replicas of tablet ac72c9fba7fe438d8fad419d2063292a
I20260929 02:59:10.084012 31852 tablet_service.cc:1514] Processing DeleteTablet for tablet ac72c9fba7fe438d8fad419d2063292a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:10 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:46314
I20260929 02:59:10.084010 31589 tablet_service.cc:1514] Processing DeleteTablet for tablet ac72c9fba7fe438d8fad419d2063292a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:10 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:57892
I20260929 02:59:10.084228 31721 tablet_service.cc:1514] Processing DeleteTablet for tablet ac72c9fba7fe438d8fad419d2063292a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:10 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:37432
I20260929 02:59:10.085218 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 31506
I20260929 02:59:10.104426 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.086048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.15.18.65:41293)
0929 02:59:10.086251 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:10.086257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:10.086409 (+ 152us) server_negotiation.cc:407] Connection header received
0929 02:59:10.086621 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:10.086626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:10.086721 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:10.086835 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:10.088723 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.090042 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.092039 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.092841 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.092972 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:10.096559 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:10.096576 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:10.096598 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:10.096638 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:10.101038 (+ 4400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:10.101972 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:10.101983 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:10.101987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:10.102176 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:10.102538 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:10.102542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:10.102546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:10.103085 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:10.103419 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:10.103727 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 02:59:10.104000 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:59:10.105762 31649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:10.106426 31956 tablet_replica.cc:330] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: stopping tablet replica
I20260929 02:59:10.106443 31957 tablet_replica.cc:330] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: stopping tablet replica
I20260929 02:59:10.106922 31956 raft_consensus.cc:2229] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:59:10.107017 31957 raft_consensus.cc:2229] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:59:10.107715 31957 raft_consensus.cc:2258] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:10.109696 31956 raft_consensus.cc:2258] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:10.110953 31957 ts_tablet_manager.cc:1905] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:59:10.112258 31956 ts_tablet_manager.cc:1905] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:59:10.118573 31957 ts_tablet_manager.cc:1918] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:59:10.118768 31957 log.cc:1196] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/ac72c9fba7fe438d8fad419d2063292a
I20260929 02:59:10.119354 31957 ts_tablet_manager.cc:1939] T ac72c9fba7fe438d8fad419d2063292a P e0e72a441d1b45769ff4341eeb4be6a1: Deleting consensus metadata
I20260929 02:59:10.122488 31453 catalog_manager.cc:4841] TS e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415): tablet ac72c9fba7fe438d8fad419d2063292a (table test-table [id=88704e2eb8784275843916231202c5cf]) successfully deleted
I20260929 02:59:10.124411 31956 ts_tablet_manager.cc:1918] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:59:10.124594 31956 log.cc:1196] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/ac72c9fba7fe438d8fad419d2063292a
I20260929 02:59:10.125335 31956 ts_tablet_manager.cc:1939] T ac72c9fba7fe438d8fad419d2063292a P 1a31d48ec1bc41dc8766d469412bfef3: Deleting consensus metadata
I20260929 02:59:10.127342 31456 catalog_manager.cc:4841] TS 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835): tablet ac72c9fba7fe438d8fad419d2063292a (table test-table [id=88704e2eb8784275843916231202c5cf]) successfully deleted
I20260929 02:59:10.214341 31955 tablet_replica.cc:330] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: stopping tablet replica
I20260929 02:59:10.214696 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 31506
W20260929 02:59:10.229054 31453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.18.65:41293 (error 108)
I20260929 02:59:10.230432 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 31658
I20260929 02:59:10.230892 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.230468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.0.0.1:57912)
0929 02:59:10.230639 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:10.230735 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:59:10.231458 31453 catalog_manager.cc:4584] TS cb465fabbbe54e20a59b2c7daba3e538 (127.15.18.65:41293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac72c9fba7fe438d8fad419d2063292a: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41293: connect: Connection refused (error 111)
I20260929 02:59:10.246991 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.231339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.15.18.66:39415)
0929 02:59:10.231552 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:10.231559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:10.231583 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:59:10.231766 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:10.231772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:10.231878 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:10.231982 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:10.234046 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.234932 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.236903 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.237297 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.237409 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:10.240869 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:10.240884 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:10.240900 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:10.240935 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:10.243908 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:10.244864 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:10.244872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:10.244874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:10.245004 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:10.245461 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:10.245464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:10.245466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:10.245862 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:10.246089 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:10.246500 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 02:59:10.246687 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:59:10.247792 31781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:10.285593 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.285049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.0.0.1:57924)
0929 02:59:10.285289 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:10.285401 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:59:10.286787 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.286393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.0.0.1:57938)
0929 02:59:10.286544 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:10.286651 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 02:59:10.335166 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 31658
I20260929 02:59:10.347556 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.347005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.0.0.1:57952)
0929 02:59:10.347272 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:10.347363 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 02:59:10.348558 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.348235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.0.0.1:57956)
0929 02:59:10.348377 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:10.348449 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 02:59:10.354918 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 31789
I20260929 02:59:10.371886 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.355809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.15.18.67:33835)
0929 02:59:10.355967 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:10.355971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:10.355991 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:59:10.356143 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:10.356147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:10.356257 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:10.356358 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:10.358423 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.359243 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.361253 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:10.361728 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:10.361839 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:10.365359 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:10.365375 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:10.365392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:10.365440 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:10.368665 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:10.369582 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:10.369589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:10.369591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:10.369780 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:10.370261 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:10.370266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:10.370268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:10.370672 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:10.370913 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:10.371300 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 02:59:10.371463 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:59:10.372702 31912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:10.429268 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.428767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.0.0.1:57960)
0929 02:59:10.429006 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:10.429114 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:59:10.430356 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.429971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.0.0.1:57972)
0929 02:59:10.430173 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:10.430244 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 02:59:10.492820 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 31789
I20260929 02:59:10.511844 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 31436
I20260929 02:59:10.513000 31499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:10.627324 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 31436
I20260929 02:59:10.627365 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:10.626829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:41293 (local address 127.0.0.1:57986)
0929 02:59:10.627064 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:10.627181 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 02:59:30.645277 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:34067
--webserver_interface=127.15.18.126
--webserver_port=39987
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:59:30.962283 31963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:30.962558 31963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:30.962671 31963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:30.970027 31963 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:59:30.970124 31963 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:59:30.970160 31963 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:59:30.970242 31963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:59:30.970319 31963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:30.970409 31963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:30.970466 31963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:59:30.970538 31963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:59:30.980435 31963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:34067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=39987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:30.981743 31963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:30.983079 31963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:30.995167 31968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:30.995195 31971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:30.995404 31969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:30.995751 31963 server_base.cc:1034] running on GCE node
I20260929 02:59:30.996569 31963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:30.997963 31963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:30.999167 31963 hybrid_clock.cc:648] HybridClock initialized: now 1790650770999096 us; error 54 us; skew 500 ppm
I20260929 02:59:30.999498 31963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:30 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650770, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:31.002662 31963 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:59:31.004176 31963 webserver.cc:460] Webserver started at http://127.15.18.126:39987/ using document root <none> and password file <none>
I20260929 02:59:31.004861 31963 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:31.004948 31963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:31.010563 31963 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:59:31.013603 31978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:31.015264 31963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:59:31.015450 31963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "9997807127644aa6b091762cc2d0f82a"
format_stamp: "Formatted at 2026-09-29 02:58:45 on dist-test-slave-q5f0"
server_key: "099f5b19c7b3226d6fa5f4932b049fe7"
server_key_iv: "d191dbe36babe2fc1bc49306313afc79"
server_key_version: "encryptionkey@0"
I20260929 02:59:31.016553 31963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:31.033608 31963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:31.038372 31963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:31.038705 31963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:31.056902 31963 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:34067
I20260929 02:59:31.056926 32029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:34067 every 8 connection(s)
I20260929 02:59:31.058499 31963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:59:31.065399 32030 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:59:31.067003 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 31963
I20260929 02:59:31.067984 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:41293
--local_ip_for_outbound_sockets=127.15.18.65
--tserver_master_addrs=127.15.18.126:34067
--webserver_port=46869
--webserver_interface=127.15.18.65
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:59:31.069635 32030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Bootstrap starting.
I20260929 02:59:31.091140 32030 log.cc:824] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:31.107430 32030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:59:31.107965 32030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Bootstrap complete.
I20260929 02:59:31.116833 32030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:59:31.117738 32030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9997807127644aa6b091762cc2d0f82a, State: Initialized, Role: FOLLOWER
I20260929 02:59:31.118360 32030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:59:31.118593 32030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:59:31.118697 32030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:59:31.118849 32030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:59:31.122938 32030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:59:31.123458 32030 leader_election.cc:302] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9997807127644aa6b091762cc2d0f82a; no voters:
I20260929 02:59:31.124089 32030 leader_election.cc:288] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 02:59:31.124230 32034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:59:31.126179 32030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:59:31.129240 32034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [term 3 LEADER]: Becoming Leader. State: Replica: 9997807127644aa6b091762cc2d0f82a, State: Running, Role: LEADER
I20260929 02:59:31.129943 32034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } }
I20260929 02:59:31.132901 32035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9997807127644aa6b091762cc2d0f82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } } }
I20260929 02:59:31.133286 32035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:31.133798 32036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9997807127644aa6b091762cc2d0f82a. Latest consensus state: current_term: 3 leader_uuid: "9997807127644aa6b091762cc2d0f82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9997807127644aa6b091762cc2d0f82a" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 34067 } } }
I20260929 02:59:31.134063 32036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:31.136068 32044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:59:31.145830 32044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:59:31.147590 32044 catalog_manager.cc:1259] Loaded cluster ID: 039f016e23fc484fa18a28ad0ff39b97
I20260929 02:59:31.147783 32044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:59:31.152263 32044 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:59:31.154173 32044 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Loaded TSK: 0
I20260929 02:59:31.154479 32044 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9997807127644aa6b091762cc2d0f82a: Loaded TSK: 1
F20260929 02:59:31.155784 32044 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f70ee1769ad google::LogMessage::Fail() at ??:0
@ 0x7f70ee17988a google::LogMessage::SendToLog() at ??:0
@ 0x7f70ee1764de google::LogMessage::Flush() at ??:0
@ 0x7f70ee177a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f70f1eda7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f70f1ec973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7103ba51c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7103ba4783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7103c302a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7103c300c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7103c7a5e9 std::function<>::operator()() at ??:0
@ 0x7f7103ba8cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7103ba6c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7103c2e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7103c2e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f70f0dc2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f70f0dc591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f70f0dc5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f70f0da0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f70f407a6db start_thread at ??:0
@ 0x7f70ed01061f clone at ??:0
W20260929 02:59:31.436151 32032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:31.436424 32032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:31.436553 32032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:31.443661 32032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:31.443773 32032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:31.443925 32032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:59:31.453513 32032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:41293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=46869
--tserver_master_addrs=127.15.18.126:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:31.454722 32032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:31.456048 32032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:31.468405 32081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:31.468459 32079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:31.468649 32078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:31.469556 32032 server_base.cc:1034] running on GCE node
I20260929 02:59:31.470139 32032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:31.471050 32032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:31.472252 32032 hybrid_clock.cc:648] HybridClock initialized: now 1790650771472201 us; error 42 us; skew 500 ppm
I20260929 02:59:31.472556 32032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:31 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650771, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:31.475469 32032 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:59:31.476902 32032 webserver.cc:460] Webserver started at http://127.15.18.65:46869/ using document root <none> and password file <none>
I20260929 02:59:31.477556 32032 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:31.477663 32032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:31.482828 32032 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:59:31.485522 32088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:31.486744 32032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:59:31.486877 32032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "cb465fabbbe54e20a59b2c7daba3e538"
format_stamp: "Formatted at 2026-09-29 02:58:46 on dist-test-slave-q5f0"
server_key: "9655e6851ee33470683e906ca5ac65d5"
server_key_iv: "6636ac715242fab654e3f03cd9d0e850"
server_key_version: "encryptionkey@0"
I20260929 02:59:31.487855 32032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:31.501751 32032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:31.505273 32032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:31.505515 32032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:31.506381 32032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:31.508081 32095 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 02:59:31.513320 32032 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 02:59:31.513430 32032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 02:59:31.513543 32032 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 02:59:31.516587 32032 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 02:59:31.516659 32032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260929 02:59:31.516992 32095 tablet_bootstrap.cc:492] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Bootstrap starting.
I20260929 02:59:31.542162 32095 log.cc:824] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:31.556070 32095 tablet_bootstrap.cc:492] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:59:31.556589 32095 tablet_bootstrap.cc:492] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Bootstrap complete.
I20260929 02:59:31.557770 32032 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:41293
I20260929 02:59:31.557869 32202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:41293 every 8 connection(s)
I20260929 02:59:31.558132 32095 ts_tablet_manager.cc:1397] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Time spent bootstrapping tablet: real 0.041s user 0.033s sys 0.005s
I20260929 02:59:31.558996 32032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:59:31.567875 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:31.562352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:57417)
0929 02:59:31.566302 (+ 3950us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:31.566607 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3644,"thread_start_us":3412,"threads_started":1}
I20260929 02:59:31.568111 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 32032
W20260929 02:59:31.568854 32203 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:34067 (0 consecutive failures): Network error: Failed to ping master at 127.15.18.126:34067: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
I20260929 02:59:31.568853 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:39415
--local_ip_for_outbound_sockets=127.15.18.66
--tserver_master_addrs=127.15.18.126:34067
--webserver_port=33499
--webserver_interface=127.15.18.66
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:59:31.567984 32095 raft_consensus.cc:348] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:31.569248 32095 raft_consensus.cc:729] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb465fabbbe54e20a59b2c7daba3e538, State: Initialized, Role: FOLLOWER
I20260929 02:59:31.569901 32095 consensus_queue.cc:260] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:31.570295 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:31.569450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:57907)
0929 02:59:31.570037 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:31.570121 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"threads_started":1}
I20260929 02:59:31.572552 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:31.571972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:34723)
0929 02:59:31.572142 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:31.572355 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 02:59:31.572871 32203 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:59:31.575551 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:31.575077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:41967)
0929 02:59:31.575278 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:31.575367 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 02:59:31.576027 32095 ts_tablet_manager.cc:1428] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538: Time spent starting tablet: real 0.018s user 0.011s sys 0.008s
I20260929 02:59:31.577078 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:31.576634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:49487)
0929 02:59:31.576868 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:31.576945 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260929 02:59:31.896637 32208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:31.896934 32208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:31.897076 32208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:31.904330 32208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:31.904459 32208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:31.904619 32208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:59:31.914464 32208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:39415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=33499
--tserver_master_addrs=127.15.18.126:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:31.915866 32208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:31.917165 32208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:31.929833 32213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:31.929768 32216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:31.929875 32214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:31.930771 32208 server_base.cc:1034] running on GCE node
I20260929 02:59:31.931409 32208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:31.932379 32208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:31.933595 32208 hybrid_clock.cc:648] HybridClock initialized: now 1790650771933538 us; error 52 us; skew 500 ppm
I20260929 02:59:31.933897 32208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:31 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650771, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:31.936883 32208 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:59:31.938262 32208 webserver.cc:460] Webserver started at http://127.15.18.66:33499/ using document root <none> and password file <none>
I20260929 02:59:31.938887 32208 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:31.938984 32208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:31.944370 32208 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:59:31.948733 32223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:31.950129 32208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:59:31.950317 32208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "e0e72a441d1b45769ff4341eeb4be6a1"
format_stamp: "Formatted at 2026-09-29 02:58:46 on dist-test-slave-q5f0"
server_key: "6358f5e6bf70708f01b81f1f8515bacc"
server_key_iv: "f6dc3eef243486c90a808453a4bc0bca"
server_key_version: "encryptionkey@0"
I20260929 02:59:31.951314 32208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:31.979589 32208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:31.983937 32208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:31.984176 32208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:31.985381 32208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:31.986822 32208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:31.986915 32208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:31.986994 32208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:31.987047 32208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:32.014807 32208 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:39415
I20260929 02:59:32.014861 32335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:39415 every 8 connection(s)
I20260929 02:59:32.016247 32208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:59:32.017431 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 32208
I20260929 02:59:32.018036 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:33835
--local_ip_for_outbound_sockets=127.15.18.67
--tserver_master_addrs=127.15.18.126:34067
--webserver_port=33767
--webserver_interface=127.15.18.67
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:59:32.025444 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.019394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:48985)
0929 02:59:32.024166 (+ 4772us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:32.024496 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4444,"thread_start_us":4207,"threads_started":1}
W20260929 02:59:32.026170 32336 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:34067 (0 consecutive failures): Network error: Failed to ping master at 127.15.18.126:34067: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
I20260929 02:59:32.026988 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.026663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:32821)
0929 02:59:32.026808 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:32.026885 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 02:59:32.027830 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.027545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:56743)
0929 02:59:32.027657 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:32.027728 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 02:59:32.028070 32336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:59:32.331444 32339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:32.331800 32339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:32.331936 32339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:32.339147 32339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:32.339264 32339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:32.339412 32339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:59:32.349052 32339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:33835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=33767
--tserver_master_addrs=127.15.18.126:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:32.350323 32339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:32.351570 32339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:32.363896 32344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:32.363863 32345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:32.363940 32347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:32.364771 32339 server_base.cc:1034] running on GCE node
I20260929 02:59:32.365340 32339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:32.366231 32339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:32.367441 32339 hybrid_clock.cc:648] HybridClock initialized: now 1790650772367384 us; error 50 us; skew 500 ppm
I20260929 02:59:32.367763 32339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:32 dist-test-slave-q5f0 krb5kdc[30885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650772, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:32.370831 32339 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:59:32.372238 32339 webserver.cc:460] Webserver started at http://127.15.18.67:33767/ using document root <none> and password file <none>
I20260929 02:59:32.372865 32339 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:32.372977 32339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:32.378700 32339 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:59:32.381481 32354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:32.382826 32339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:59:32.382979 32339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "1a31d48ec1bc41dc8766d469412bfef3"
format_stamp: "Formatted at 2026-09-29 02:58:47 on dist-test-slave-q5f0"
server_key: "801833e89d12ae83d5f8212019f8bcfc"
server_key_iv: "0359150d0df66a7337616fe8d0465669"
server_key_version: "encryptionkey@0"
I20260929 02:59:32.383951 32339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:32.400039 32339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:32.403945 32339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:32.404193 32339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:32.405114 32339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:32.406397 32339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:32.406493 32339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:32.406595 32339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:32.406666 32339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:32.436110 32339 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:33835
I20260929 02:59:32.436158 32466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:33835 every 8 connection(s)
I20260929 02:59:32.437459 32339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:59:32.439853 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 32339
I20260929 02:59:32.442227 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.440263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:46675)
0929 02:59:32.441079 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:32.441373 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":301,"threads_started":1}
W20260929 02:59:32.442832 32467 heartbeater.cc:644] Failed to heartbeat to 127.15.18.126:34067 (0 consecutive failures): Network error: Failed to ping master at 127.15.18.126:34067: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
I20260929 02:59:32.443537 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.443238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:50561)
0929 02:59:32.443355 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:32.443416 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 02:59:32.444227 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.443969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:58583)
0929 02:59:32.444066 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:32.444131 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 02:59:32.444417 32467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:59:32.578657 32471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.577901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:38247)
0929 02:59:32.578348 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:32.578444 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":208,"threads_started":1}
I20260929 02:59:32.880724 32472 raft_consensus.cc:482] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:59:32.881011 32472 raft_consensus.cc:504] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:32.882557 32472 leader_election.cc:288] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835), e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415)
Sep 29 02:59:32 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650771, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
Sep 29 02:59:32 dist-test-slave-q5f0 krb5kdc[30885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650771, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:59:32.909773 32471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.882868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:33835 (local address 127.15.18.65:33243)
0929 02:59:32.883212 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:32.883273 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:32.883948 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:32.888666 (+ 4718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:32.888679 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:32.889700 (+ 1021us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:32.890796 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:32.890833 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:32.893182 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:32.893186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:32.895340 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:32.895357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:32.896853 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:32.896860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:32.897234 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:32.898108 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:32.898149 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:32.901385 (+ 3236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:32.905377 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:32.905387 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:32.905405 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:32.906199 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:32.906688 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:32.906694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:32.906697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:32.907009 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:32.908006 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:32.908014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:32.908501 (+ 487us) client_negotiation.cc:769] Sending connection context
0929 02:59:32.908823 (+ 322us) client_negotiation.cc:240] Negotiation successful
0929 02:59:32.909377 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":353,"spinlock_wait_cycles":60672}
I20260929 02:59:32.909960 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.883219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:33243 (local address 127.15.18.67:33835)
0929 02:59:32.885119 (+ 1900us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:32.885137 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:32.885188 (+ 51us) server_negotiation.cc:407] Connection header received
0929 02:59:32.887491 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:32.887502 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:32.888068 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:32.888459 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:32.891053 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:32.892980 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:32.895539 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:32.896528 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:32.896718 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:32.901646 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:32.901656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:32.901669 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:32.901714 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:32.905195 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:32.906335 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:32.906344 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:32.906351 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:32.906571 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:32.907121 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:32.907126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:32.907130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:32.907854 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:32.908022 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:32.909102 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0929 02:59:32.909574 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1698,"thread_start_us":1562,"threads_started":1}
I20260929 02:59:32.910445 32473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.882915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:39415 (local address 127.15.18.65:44403)
0929 02:59:32.884743 (+ 1828us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:32.884764 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:32.884885 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:32.888666 (+ 3781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:32.888679 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:32.889700 (+ 1021us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:32.890796 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:32.890826 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:32.893179 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:32.893185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:32.895344 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:32.895357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:32.896853 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:32.896860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:32.897228 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:32.898108 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:32.898149 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:32.901162 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:32.905377 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:32.905387 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:32.905405 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:32.906199 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:32.906688 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:32.906694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:32.906697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:32.907009 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:32.908006 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:32.908014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:32.908501 (+ 487us) client_negotiation.cc:769] Sending connection context
0929 02:59:32.908823 (+ 322us) client_negotiation.cc:240] Negotiation successful
0929 02:59:32.910077 (+ 1254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"mutex_wait_us":1090,"spinlock_wait_cycles":63232,"thread_start_us":1548,"threads_started":1}
I20260929 02:59:32.910609 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:32.883147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:44403 (local address 127.15.18.66:39415)
0929 02:59:32.885376 (+ 2229us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:32.885390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:32.885426 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:59:32.887475 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:32.887489 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:32.888068 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:32.888459 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:32.891053 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:32.892980 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:32.895533 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:32.896528 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:32.896718 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:32.901462 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:32.901473 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:32.901493 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:32.901550 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:32.905195 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:32.906341 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:32.906348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:32.906354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:32.906587 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:32.907121 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:32.907126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:32.907130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:32.907849 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:32.908022 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:32.909102 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0929 02:59:32.909566 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2063,"thread_start_us":1927,"threads_started":1}
W20260929 02:59:32.914100 32089 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
W20260929 02:59:32.914393 32092 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
I20260929 02:59:32.914523 32092 leader_election.cc:302] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb465fabbbe54e20a59b2c7daba3e538; no voters: 1a31d48ec1bc41dc8766d469412bfef3, e0e72a441d1b45769ff4341eeb4be6a1
I20260929 02:59:32.914938 32472 raft_consensus.cc:2726] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:59:33.029480 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:33.028597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:47149)
0929 02:59:33.029071 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:33.029220 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":229,"threads_started":1}
I20260929 02:59:33.445675 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:33.444969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:55333)
0929 02:59:33.445388 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:33.445496 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":207,"threads_started":1}
I20260929 02:59:33.580483 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:33.579707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:34809)
0929 02:59:33.580177 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:33.580270 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
I20260929 02:59:34.031170 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:34.030350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:53813)
0929 02:59:34.030858 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:34.030968 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":283,"threads_started":1}
I20260929 02:59:34.447293 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:34.446543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:40387)
0929 02:59:34.447019 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:34.447109 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":268,"threads_started":1}
I20260929 02:59:34.582336 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:34.581466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:33571)
0929 02:59:34.582075 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:34.582159 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":321,"threads_started":1}
I20260929 02:59:34.917034 32482 raft_consensus.cc:482] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:59:34.917232 32482 raft_consensus.cc:504] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:34.917881 32482 leader_election.cc:288] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835), e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415)
W20260929 02:59:34.918890 32092 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
W20260929 02:59:34.919051 32089 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
I20260929 02:59:34.919164 32089 leader_election.cc:302] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb465fabbbe54e20a59b2c7daba3e538; no voters: 1a31d48ec1bc41dc8766d469412bfef3, e0e72a441d1b45769ff4341eeb4be6a1
I20260929 02:59:34.919402 32482 raft_consensus.cc:2726] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:59:35.032895 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:35.032096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:56205)
0929 02:59:35.032634 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:35.032727 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260929 02:59:35.448745 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:35.448068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:46089)
0929 02:59:35.448505 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:35.448591 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":224,"threads_started":1}
I20260929 02:59:35.584012 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:35.583241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:39341)
0929 02:59:35.583745 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:35.583849 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":296,"threads_started":1}
I20260929 02:59:36.034466 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:36.033758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:48811)
0929 02:59:36.034194 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:36.034271 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260929 02:59:36.450413 32487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:36.449662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:48921)
0929 02:59:36.450122 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:36.450209 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":217,"threads_started":1}
I20260929 02:59:36.585886 32488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:36.585128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:50031)
0929 02:59:36.585592 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:36.585676 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":242,"threads_started":1}
I20260929 02:59:37.036216 32489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:37.035374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:33899)
0929 02:59:37.035914 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:37.036023 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":275,"threads_started":1}
I20260929 02:59:37.452006 32490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:37.451300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:40603)
0929 02:59:37.451754 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:37.451848 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":258,"threads_started":1}
I20260929 02:59:37.587602 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:37.586841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:52073)
0929 02:59:37.587304 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:37.587396 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260929 02:59:37.770720 32492 raft_consensus.cc:482] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:59:37.770931 32492 raft_consensus.cc:504] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:37.771557 32492 leader_election.cc:288] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835), e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415)
W20260929 02:59:37.772550 32092 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
W20260929 02:59:37.772768 32089 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
I20260929 02:59:37.772923 32089 leader_election.cc:302] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb465fabbbe54e20a59b2c7daba3e538; no voters: 1a31d48ec1bc41dc8766d469412bfef3, e0e72a441d1b45769ff4341eeb4be6a1
I20260929 02:59:37.773169 32492 raft_consensus.cc:2726] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:59:38.037855 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:38.037119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:52709)
0929 02:59:38.037571 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:38.037677 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":232,"threads_started":1}
I20260929 02:59:38.453662 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:38.452791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:39057)
0929 02:59:38.453293 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:38.453417 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":263,"threads_started":1}
I20260929 02:59:38.589154 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:38.588450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:45369)
0929 02:59:38.588876 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:38.588958 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
I20260929 02:59:39.039695 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:39.038738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:43529)
0929 02:59:39.039329 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:39.039457 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":278,"threads_started":1}
I20260929 02:59:39.383081 32497 raft_consensus.cc:482] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:59:39.383282 32497 raft_consensus.cc:504] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:39.383963 32497 leader_election.cc:288] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835), e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415)
W20260929 02:59:39.384855 32092 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
W20260929 02:59:39.385007 32089 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
I20260929 02:59:39.385118 32089 leader_election.cc:302] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb465fabbbe54e20a59b2c7daba3e538; no voters: 1a31d48ec1bc41dc8766d469412bfef3, e0e72a441d1b45769ff4341eeb4be6a1
I20260929 02:59:39.385341 32497 raft_consensus.cc:2726] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:59:39.455607 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:39.454769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:49519)
0929 02:59:39.455268 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:39.455355 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":222,"threads_started":1}
I20260929 02:59:39.590813 32499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:39.590050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:46927)
0929 02:59:39.590498 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:39.590628 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260929 02:59:40.041221 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:40.040534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:53671)
0929 02:59:40.040961 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:40.041039 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":215,"threads_started":1}
I20260929 02:59:40.457193 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:40.456471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:38259)
0929 02:59:40.456920 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:40.457009 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":228,"threads_started":1}
I20260929 02:59:40.592562 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:40.591797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:38711)
0929 02:59:40.592288 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:40.592382 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":257,"threads_started":1}
I20260929 02:59:41.042886 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:41.042148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:42533)
0929 02:59:41.042621 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:41.042707 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260929 02:59:41.458953 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:41.458161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:35093)
0929 02:59:41.458681 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:41.458799 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":261,"threads_started":1}
I20260929 02:59:41.594177 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:41.593452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:52281)
0929 02:59:41.593914 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:41.594001 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":256,"threads_started":1}
I20260929 02:59:42.044637 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:42.043875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:55485)
0929 02:59:42.044366 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:42.044455 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":245,"threads_started":1}
I20260929 02:59:42.460464 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:42.459812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:42433)
0929 02:59:42.460204 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:42.460282 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":200,"threads_started":1}
I20260929 02:59:42.595798 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:42.595008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:46871)
0929 02:59:42.595464 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:42.595590 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":244,"threads_started":1}
I20260929 02:59:43.046499 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:43.045598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:58285)
0929 02:59:43.046230 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:43.046327 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":362,"threads_started":1}
I20260929 02:59:43.063192 32510 raft_consensus.cc:482] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:59:43.063373 32510 raft_consensus.cc:504] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:43.064080 32510 leader_election.cc:288] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835), e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415)
W20260929 02:59:43.065027 32089 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
W20260929 02:59:43.065189 32092 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
I20260929 02:59:43.065297 32092 leader_election.cc:302] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb465fabbbe54e20a59b2c7daba3e538; no voters: 1a31d48ec1bc41dc8766d469412bfef3, e0e72a441d1b45769ff4341eeb4be6a1
I20260929 02:59:43.065521 32510 raft_consensus.cc:2726] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:59:43.462035 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:43.461275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:50997)
0929 02:59:43.461789 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:43.461872 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":251,"threads_started":1}
I20260929 02:59:43.597280 32512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:43.596603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:40277)
0929 02:59:43.597039 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:43.597121 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
I20260929 02:59:44.048110 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:44.047397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:36341)
0929 02:59:44.047855 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:44.047938 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":252,"threads_started":1}
I20260929 02:59:44.463698 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:44.462903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:46153)
0929 02:59:44.463383 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:44.463472 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":258,"threads_started":1}
I20260929 02:59:44.599107 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:44.598310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:51269)
0929 02:59:44.598825 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:44.598928 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":236,"threads_started":1}
I20260929 02:59:45.049687 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:45.048943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:38353)
0929 02:59:45.049416 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:45.049516 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":230,"threads_started":1}
I20260929 02:59:45.465304 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:45.464595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:33161)
0929 02:59:45.465042 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:45.465140 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":224,"threads_started":1}
I20260929 02:59:45.600872 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:45.600069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:35757)
0929 02:59:45.600582 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:45.600670 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":283,"threads_started":1}
I20260929 02:59:46.051380 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:46.050578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:38575)
0929 02:59:46.051093 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:46.051178 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":298,"threads_started":1}
I20260929 02:59:46.055766 32520 raft_consensus.cc:482] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:59:46.055945 32520 raft_consensus.cc:504] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb465fabbbe54e20a59b2c7daba3e538" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 41293 } } peers { permanent_uuid: "1a31d48ec1bc41dc8766d469412bfef3" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33835 } } peers { permanent_uuid: "e0e72a441d1b45769ff4341eeb4be6a1" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 39415 } }
I20260929 02:59:46.056559 32520 leader_election.cc:288] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835), e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415)
W20260929 02:59:46.057478 32089 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0e72a441d1b45769ff4341eeb4be6a1 (127.15.18.66:39415): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
W20260929 02:59:46.057623 32092 leader_election.cc:341] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1a31d48ec1bc41dc8766d469412bfef3 (127.15.18.67:33835): Not found: Tablet not found: ac72c9fba7fe438d8fad419d2063292a
I20260929 02:59:46.057724 32092 leader_election.cc:302] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb465fabbbe54e20a59b2c7daba3e538; no voters: 1a31d48ec1bc41dc8766d469412bfef3, e0e72a441d1b45769ff4341eeb4be6a1
I20260929 02:59:46.057940 32520 raft_consensus.cc:2726] T ac72c9fba7fe438d8fad419d2063292a P cb465fabbbe54e20a59b2c7daba3e538 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:59:46.467059 32521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:46.466228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:34135)
0929 02:59:46.466754 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:46.466848 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":243,"threads_started":1}
I20260929 02:59:46.602897 32522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:46.602158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.65:35779)
0929 02:59:46.602623 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:46.602723 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
I20260929 02:59:47.053265 32523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:47.052493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.66:49545)
0929 02:59:47.052963 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:47.053080 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":238,"threads_started":1}
I20260929 02:59:47.448323 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 32032
I20260929 02:59:47.463044 32524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:47.449343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.15.18.65:41293)
0929 02:59:47.450412 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:47.450422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:47.450454 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:59:47.450534 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:47.450543 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:47.450861 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:47.451007 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:47.452402 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:47.453343 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:47.454544 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:47.455166 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:47.455256 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:47.457732 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:47.457744 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:47.457758 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:47.457793 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:47.460676 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:47.461294 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:47.461300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:47.461302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:47.461422 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:47.461716 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:47.461719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:47.461721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:47.462083 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:47.462315 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:47.462519 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 02:59:47.462714 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
I20260929 02:59:47.464336 32198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:47.469115 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:47.468293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:34067 (local address 127.15.18.67:53247)
0929 02:59:47.468783 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:47.468918 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.126:34067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":233,"threads_started":1}
I20260929 02:59:47.553650 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 32032
I20260929 02:59:47.567756 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 32208
I20260929 02:59:47.580303 32527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:47.568545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.15.18.66:39415)
0929 02:59:47.568997 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:47.569001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:47.569019 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:59:47.569074 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:47.569078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:47.569137 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:47.569246 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:47.570558 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:47.571339 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:47.572735 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:47.573138 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:47.573237 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:47.575612 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:47.575617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:47.575620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:47.575651 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:47.578150 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:47.578694 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:47.578697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:47.578699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:47.578832 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:47.579105 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:47.579108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:47.579109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:47.579370 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:47.579685 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:47.579790 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 02:59:47.579951 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":243,"threads_started":1}
I20260929 02:59:47.580948 32331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:47.664088 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 32208
I20260929 02:59:47.678267 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 32339
I20260929 02:59:47.691401 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:47.679124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.15.18.67:33835)
0929 02:59:47.679719 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:47.679724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:47.679740 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:59:47.679793 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:47.679796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:47.679884 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:47.679970 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:47.681306 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:47.682074 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:47.683362 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:47.683826 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:47.683924 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:47.686395 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:47.686399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:47.686402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:47.686431 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:47.689134 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:47.689785 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:47.689788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:47.689791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:47.689929 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:47.690219 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:47.690221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:47.690223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:47.690473 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:47.690667 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:47.690905 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 02:59:47.691060 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":275,"threads_started":1}
I20260929 02:59:47.692229 32462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:47.774016 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 32339
2026-09-29T02:59:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63662 ms)
[----------] 30 tests from SecurityITest (290909 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:59:47 dist-test-slave-q5f0 krb5kdc[32533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:59:47 dist-test-slave-q5f0 krb5kdc[32533](info): set up 2 sockets
Sep 29 02:59:47 dist-test-slave-q5f0 krb5kdc[32533](info): commencing operation
krb5kdc: starting...
W20260929 02:59:48.783802 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:59:48 dist-test-slave-q5f0 krb5kdc[32533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:59:49.029134 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:44147
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:44147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:59:49.363117 32546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:49.363417 32546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:49.363583 32546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:49.373255 32546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:59:49.373385 32546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:49.373462 32546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:49.373540 32546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:59:49.373600 32546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:59:49.383610 32546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:44147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:44147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:49.384930 32546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:49.386243 32546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:49.398262 32551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:49.398300 32552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:49.398227 32554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:49.398522 32546 server_base.cc:1034] running on GCE node
I20260929 02:59:49.399653 32546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:49.401078 32546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:49.402318 32546 hybrid_clock.cc:648] HybridClock initialized: now 1790650789402273 us; error 47 us; skew 500 ppm
I20260929 02:59:49.402662 32546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:49 dist-test-slave-q5f0 krb5kdc[32533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650789, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:49.405750 32546 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:59:49.407186 32546 webserver.cc:460] Webserver started at http://127.15.18.126:36493/ using document root <none> and password file <none>
I20260929 02:59:49.407908 32546 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:49.408027 32546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:49.408316 32546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:49.411321 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "ae0598cf20b0414fa2d6ce57d8ac773d"
format_stamp: "Formatted at 2026-09-29 02:59:49 on dist-test-slave-q5f0"
server_key: "7f3fdc485ead6b6b524affff6c945c76"
server_key_iv: "0e7fd5a8497834ca5fec8a4171926349"
server_key_version: "encryptionkey@0"
I20260929 02:59:49.412107 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "ae0598cf20b0414fa2d6ce57d8ac773d"
format_stamp: "Formatted at 2026-09-29 02:59:49 on dist-test-slave-q5f0"
server_key: "7f3fdc485ead6b6b524affff6c945c76"
server_key_iv: "0e7fd5a8497834ca5fec8a4171926349"
server_key_version: "encryptionkey@0"
I20260929 02:59:49.417117 32546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:59:49.420419 32561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:49.421710 32546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:59:49.421871 32546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "ae0598cf20b0414fa2d6ce57d8ac773d"
format_stamp: "Formatted at 2026-09-29 02:59:49 on dist-test-slave-q5f0"
server_key: "7f3fdc485ead6b6b524affff6c945c76"
server_key_iv: "0e7fd5a8497834ca5fec8a4171926349"
server_key_version: "encryptionkey@0"
I20260929 02:59:49.422049 32546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:49.436685 32546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:49.440475 32546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:49.440748 32546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:49.457520 32546 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:44147
I20260929 02:59:49.457538 32612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:44147 every 8 connection(s)
I20260929 02:59:49.458869 32546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:59:49.461601 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 32546
I20260929 02:59:49.461824 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:59:49.462622 15433 external_mini_cluster.cc:1428] Setting key 5515f662748741417860d5d546be765c
I20260929 02:59:49.463157 32613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:49.475442 32613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d: Bootstrap starting.
Sep 29 02:59:49 dist-test-slave-q5f0 krb5kdc[32533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:49.479077 32613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:49.480499 32613 log.cc:824] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:49.483800 32613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d: No bootstrap required, opened a new log
I20260929 02:59:49.486320 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:49.464735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.15.18.126:44147)
0929 02:59:49.465503 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:49.465514 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:49.465553 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:59:49.467787 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:49.467810 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:49.468224 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:49.468602 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:49.470065 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:49.471336 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:49.472672 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:49.473401 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:49.473583 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:49.477752 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:49.477764 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:49.477785 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:49.477841 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:49.481669 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:49.482453 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:49.482463 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:49.482471 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:49.482673 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:49.483101 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:49.483106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:49.483109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:49.483851 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:49.484145 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:49.484535 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 02:59:49.484987 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":346,"threads_started":1}
I20260929 02:59:49.492383 32613 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0598cf20b0414fa2d6ce57d8ac773d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44147 } }
I20260929 02:59:49.492715 32613 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:49.492787 32613 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0598cf20b0414fa2d6ce57d8ac773d, State: Initialized, Role: FOLLOWER
I20260929 02:59:49.493300 32613 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0598cf20b0414fa2d6ce57d8ac773d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44147 } }
I20260929 02:59:49.493503 32613 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:59:49.493597 32613 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:59:49.493716 32613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:49.496004 32613 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0598cf20b0414fa2d6ce57d8ac773d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44147 } }
I20260929 02:59:49.496441 32613 leader_election.cc:302] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae0598cf20b0414fa2d6ce57d8ac773d; no voters:
I20260929 02:59:49.496992 32613 leader_election.cc:288] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:59:49.497129 32618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:59:49.497710 32618 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [term 1 LEADER]: Becoming Leader. State: Replica: ae0598cf20b0414fa2d6ce57d8ac773d, State: Running, Role: LEADER
I20260929 02:59:49.498220 32618 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0598cf20b0414fa2d6ce57d8ac773d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44147 } }
I20260929 02:59:49.498910 32613 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:59:49.500646 32619 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae0598cf20b0414fa2d6ce57d8ac773d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0598cf20b0414fa2d6ce57d8ac773d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44147 } } }
I20260929 02:59:49.501072 32619 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:49.502604 32620 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae0598cf20b0414fa2d6ce57d8ac773d. Latest consensus state: current_term: 1 leader_uuid: "ae0598cf20b0414fa2d6ce57d8ac773d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae0598cf20b0414fa2d6ce57d8ac773d" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 44147 } } }
I20260929 02:59:49.503198 32620 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:49.503824 32626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:59:49.508911 32626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:59:49.518057 32626 catalog_manager.cc:1347] Generated new cluster ID: 5a89bc83b0c84a34bda00162157980c7
I20260929 02:59:49.518180 32626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:59:49.565335 32626 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:59:49.566792 32626 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:59:49.578070 32626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d: Generated new TSK 0
I20260929 02:59:49.578883 32626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:59:49.639241 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:44147
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:59:49.956198 32639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:49.956466 32639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:49.956594 32639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:49.963716 32639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:49.963833 32639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:49.963980 32639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:59:49.973565 32639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:49.974802 32639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:49.976063 32639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:49.988503 32645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:49.988529 32644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:49.988785 32647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:49.989138 32639 server_base.cc:1034] running on GCE node
I20260929 02:59:49.989761 32639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:49.990656 32639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:49.991852 32639 hybrid_clock.cc:648] HybridClock initialized: now 1790650789991803 us; error 41 us; skew 500 ppm
I20260929 02:59:49.992151 32639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:49 dist-test-slave-q5f0 krb5kdc[32533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650789, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:49.995069 32639 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:59:49.996459 32639 webserver.cc:460] Webserver started at http://127.15.18.65:36549/ using document root <none> and password file <none>
I20260929 02:59:49.997107 32639 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:49.997226 32639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:49.997522 32639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:50.000540 32639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5"
format_stamp: "Formatted at 2026-09-29 02:59:49 on dist-test-slave-q5f0"
server_key: "e6b2624f7271e58d00b2d2fbc23f04f0"
server_key_iv: "e7f05a1c5d89ea95b7b67913c8150429"
server_key_version: "encryptionkey@0"
I20260929 02:59:50.001274 32639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5"
format_stamp: "Formatted at 2026-09-29 02:59:49 on dist-test-slave-q5f0"
server_key: "e6b2624f7271e58d00b2d2fbc23f04f0"
server_key_iv: "e7f05a1c5d89ea95b7b67913c8150429"
server_key_version: "encryptionkey@0"
I20260929 02:59:50.006392 32639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:59:50.009934 32654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:50.011277 32639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:59:50.011435 32639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5"
format_stamp: "Formatted at 2026-09-29 02:59:49 on dist-test-slave-q5f0"
server_key: "e6b2624f7271e58d00b2d2fbc23f04f0"
server_key_iv: "e7f05a1c5d89ea95b7b67913c8150429"
server_key_version: "encryptionkey@0"
I20260929 02:59:50.011647 32639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:50.026928 32639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:50.031066 32639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:50.031337 32639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:50.032358 32639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:50.033670 32639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:50.033746 32639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:50.033869 32639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:50.033928 32639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:50.062749 32639 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:36517
I20260929 02:59:50.062789 32766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:36517 every 8 connection(s)
I20260929 02:59:50.064047 32639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:59:50.071813 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 32639
I20260929 02:59:50.072027 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:59:50.072691 15433 external_mini_cluster.cc:1428] Setting key cc984865585bcfa72a98f8d1e8152eda
Sep 29 02:59:50 dist-test-slave-q5f0 krb5kdc[32533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650789, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:50.084290 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:50.066402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:41511 (local address 127.15.18.126:44147)
0929 02:59:50.066829 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:50.066835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:50.067809 (+ 974us) server_negotiation.cc:407] Connection header received
0929 02:59:50.069552 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:50.069557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:50.069625 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:50.069718 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:50.071165 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:50.072184 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:50.073822 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:50.074420 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:50.074541 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:50.078529 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:50.078534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:50.078538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:50.078569 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:50.081056 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:50.081843 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:50.081847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:50.081850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:50.081983 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:50.082425 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:50.082428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:50.082431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:50.082803 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:50.082972 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:50.083843 (+ 871us) server_negotiation.cc:299] Negotiation successful
0929 02:59:50.084035 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":188,"threads_started":1}
I20260929 02:59:50.085578 302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:50.066937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44147 (local address 127.15.18.65:41511)
0929 02:59:50.067652 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:50.067702 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:50.069299 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:50.069903 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:50.069911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:50.070406 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:50.071004 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:50.071026 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:50.072329 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:50.072333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:50.073696 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:50.073708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:50.074620 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:50.074627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:50.074912 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:50.075606 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:50.075633 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:50.078295 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:50.081191 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:50.081199 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:50.081215 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:50.081747 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:50.082077 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:50.082081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:50.082085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:50.082333 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:50.082951 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:50.082960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:50.083406 (+ 446us) client_negotiation.cc:769] Sending connection context
0929 02:59:50.083737 (+ 331us) client_negotiation.cc:240] Negotiation successful
0929 02:59:50.084262 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":271,"threads_started":1}
I20260929 02:59:50.087134 32767 heartbeater.cc:347] Connected to a master server at 127.15.18.126:44147
I20260929 02:59:50.087451 32767 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:50.088197 32767 heartbeater.cc:511] Master 127.15.18.126:44147 requested a full tablet report, sending...
I20260929 02:59:50.090209 32578 ts_manager.cc:194] Registered new tserver with Master: dd5237f7dcc24a3db57a75c4a14d1ee5 (127.15.18.65:36517)
I20260929 02:59:50.091722 32578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:41511
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:59:50.124842 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:44147
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:59:50.439734 305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:50.440009 305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:50.440100 305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:50.447242 305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:50.447355 305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:50.447465 305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:59:50.457520 305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:50.459064 305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:50.460662 305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:50.475596 311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:50.475698 313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:50.475821 310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:50.476547 305 server_base.cc:1034] running on GCE node
I20260929 02:59:50.477149 305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:50.478114 305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:50.479336 305 hybrid_clock.cc:648] HybridClock initialized: now 1790650790479293 us; error 43 us; skew 500 ppm
I20260929 02:59:50.479691 305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:50 dist-test-slave-q5f0 krb5kdc[32533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:50.482853 305 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:59:50.484447 305 webserver.cc:460] Webserver started at http://127.15.18.66:45515/ using document root <none> and password file <none>
I20260929 02:59:50.485136 305 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:50.485255 305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:50.485558 305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:50.488492 305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "6d280df9f7094275aec3fda15c8e3ade"
format_stamp: "Formatted at 2026-09-29 02:59:50 on dist-test-slave-q5f0"
server_key: "ac625ca44b1ae9d76c2aeb30f0e9e007"
server_key_iv: "5b01136f7b66463a7aaefa2c52a3055f"
server_key_version: "encryptionkey@0"
I20260929 02:59:50.489252 305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "6d280df9f7094275aec3fda15c8e3ade"
format_stamp: "Formatted at 2026-09-29 02:59:50 on dist-test-slave-q5f0"
server_key: "ac625ca44b1ae9d76c2aeb30f0e9e007"
server_key_iv: "5b01136f7b66463a7aaefa2c52a3055f"
server_key_version: "encryptionkey@0"
I20260929 02:59:50.494546 305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:59:50.498178 320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:50.499640 305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:59:50.499817 305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "6d280df9f7094275aec3fda15c8e3ade"
format_stamp: "Formatted at 2026-09-29 02:59:50 on dist-test-slave-q5f0"
server_key: "ac625ca44b1ae9d76c2aeb30f0e9e007"
server_key_iv: "5b01136f7b66463a7aaefa2c52a3055f"
server_key_version: "encryptionkey@0"
I20260929 02:59:50.500029 305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:50.514258 305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:50.518891 305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:50.519220 305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:50.520294 305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:50.521780 305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:50.521867 305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:50.521999 305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:50.522075 305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:50.553442 305 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:43913
I20260929 02:59:50.553465 433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:43913 every 8 connection(s)
I20260929 02:59:50.554796 305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:59:50.557029 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 305
I20260929 02:59:50.557210 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:59:50.557766 15433 external_mini_cluster.cc:1428] Setting key 8648768e6130c3fd4600c11adac3ca2d
Sep 29 02:59:50 dist-test-slave-q5f0 krb5kdc[32533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:50.575075 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:50.557403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:32861 (local address 127.15.18.126:44147)
0929 02:59:50.557625 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:50.557632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:50.559499 (+ 1867us) server_negotiation.cc:407] Connection header received
0929 02:59:50.560722 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:50.560729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:50.560829 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:50.560943 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:50.562362 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:50.563358 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:50.564961 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:50.565546 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:50.565677 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:50.569113 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:50.569121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:50.569126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:50.569165 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:50.572168 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:50.572854 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:50.572859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:50.572863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:50.573022 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:50.573405 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:50.573409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:50.573411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:50.573761 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:50.573902 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:50.574543 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 02:59:50.574762 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:59:50.576040 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:50.557857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44147 (local address 127.15.18.66:32861)
0929 02:59:50.558652 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:50.558705 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:50.560439 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:50.561112 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:50.561120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:50.561634 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:50.562192 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:50.562211 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:50.563551 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:50.563557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:50.564810 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:50.564823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:50.565769 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:50.565774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:50.565973 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:50.566617 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:50.566640 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:50.568883 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:50.572321 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:50.572328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:50.572341 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:50.572742 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:50.573111 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:50.573114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:50.573116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:50.573305 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:50.573865 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:50.573870 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:50.574187 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 02:59:50.574419 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 02:59:50.574808 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":309,"threads_started":1}
I20260929 02:59:50.577442 434 heartbeater.cc:347] Connected to a master server at 127.15.18.126:44147
I20260929 02:59:50.577762 434 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:50.578645 434 heartbeater.cc:511] Master 127.15.18.126:44147 requested a full tablet report, sending...
I20260929 02:59:50.580314 32578 ts_manager.cc:194] Registered new tserver with Master: 6d280df9f7094275aec3fda15c8e3ade (127.15.18.66:43913)
I20260929 02:59:50.581085 32578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:32861
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:59:50.611894 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:44147
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:59:50.924916 439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:50.925199 439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:50.925341 439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:50.932493 439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:50.932611 439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:50.932760 439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:59:50.942283 439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:44147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:50.943578 439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:50.944855 439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:50.957736 444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:50.957787 445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:50.957821 447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:50.958487 439 server_base.cc:1034] running on GCE node
I20260929 02:59:50.959187 439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:50.960134 439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:50.961341 439 hybrid_clock.cc:648] HybridClock initialized: now 1790650790961285 us; error 51 us; skew 500 ppm
I20260929 02:59:50.961642 439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:50 dist-test-slave-q5f0 krb5kdc[32533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:50.964674 439 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:59:50.966037 439 webserver.cc:460] Webserver started at http://127.15.18.67:41155/ using document root <none> and password file <none>
I20260929 02:59:50.966687 439 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:50.966804 439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:50.967084 439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:50.969883 439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "40ce0e0bfef245ffb5b1748447101717"
format_stamp: "Formatted at 2026-09-29 02:59:50 on dist-test-slave-q5f0"
server_key: "f0a77bbb68f1d1f66dd8528b7c6b47bb"
server_key_iv: "46dff451fc8ba97f117761fe80aaa063"
server_key_version: "encryptionkey@0"
I20260929 02:59:50.970590 439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "40ce0e0bfef245ffb5b1748447101717"
format_stamp: "Formatted at 2026-09-29 02:59:50 on dist-test-slave-q5f0"
server_key: "f0a77bbb68f1d1f66dd8528b7c6b47bb"
server_key_iv: "46dff451fc8ba97f117761fe80aaa063"
server_key_version: "encryptionkey@0"
I20260929 02:59:50.975617 439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:59:50.978940 454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:50.980316 439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:59:50.980463 439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "40ce0e0bfef245ffb5b1748447101717"
format_stamp: "Formatted at 2026-09-29 02:59:50 on dist-test-slave-q5f0"
server_key: "f0a77bbb68f1d1f66dd8528b7c6b47bb"
server_key_iv: "46dff451fc8ba97f117761fe80aaa063"
server_key_version: "encryptionkey@0"
I20260929 02:59:50.980633 439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:50.999442 439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:51.003773 439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:51.004063 439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:51.005048 439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:51.006363 439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:51.006439 439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:51.006539 439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:51.006592 439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:51.034919 439 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:36257
I20260929 02:59:51.034960 567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:36257 every 8 connection(s)
I20260929 02:59:51.036216 439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:59:51.044663 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 439
I20260929 02:59:51.044844 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:59:51.045460 15433 external_mini_cluster.cc:1428] Setting key da8d519142dbfbdc47f278a156416d91
Sep 29 02:59:51 dist-test-slave-q5f0 krb5kdc[32533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650790, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:51.055215 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.038650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:54855 (local address 127.15.18.126:44147)
0929 02:59:51.038877 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.038883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.040076 (+ 1193us) server_negotiation.cc:407] Connection header received
0929 02:59:51.041801 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.041805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.041876 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.041967 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:51.043353 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.044173 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.046282 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.046851 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.046966 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.050533 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:51.050539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:51.050542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:51.050575 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:51.052864 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:51.053474 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:51.053477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:51.053480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:51.053592 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:51.053911 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:51.053914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:51.053916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:51.054159 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:51.054264 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.054825 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.054982 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 02:59:51.056169 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.039186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:44147 (local address 127.15.18.67:54855)
0929 02:59:51.039923 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:51.039967 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:51.041549 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:51.042147 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:51.042154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:51.042663 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:51.043194 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.043211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.044297 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.044302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.046130 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.046141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.047096 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.047104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.047372 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.048103 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:51.048129 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:51.050342 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:51.052980 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:51.052986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:51.052998 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:51.053390 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:51.053666 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:51.053668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:51.053671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:51.053840 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:51.054251 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:51.054256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:51.054533 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 02:59:51.054762 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 02:59:51.055112 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":312,"threads_started":1}
I20260929 02:59:51.057433 568 heartbeater.cc:347] Connected to a master server at 127.15.18.126:44147
I20260929 02:59:51.057710 568 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:51.058395 568 heartbeater.cc:511] Master 127.15.18.126:44147 requested a full tablet report, sending...
I20260929 02:59:51.059828 32578 ts_manager.cc:194] Registered new tserver with Master: 40ce0e0bfef245ffb5b1748447101717 (127.15.18.67:36257)
I20260929 02:59:51.060801 32578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:54855
I20260929 02:59:51.061190 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:59:51.078924 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.065923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.15.18.126:44147)
0929 02:59:51.066088 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.066092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.066542 (+ 450us) server_negotiation.cc:407] Connection header received
0929 02:59:51.066716 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.066723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.066802 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.066890 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:51.068560 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.069304 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.070688 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.071132 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.071262 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.073769 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:51.073775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:51.073778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:51.073813 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:51.076366 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:51.077073 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:51.077079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:51.077083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:51.077214 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:51.077552 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:51.077558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:51.077561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:51.077876 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:51.078103 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.078404 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.078606 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:59:51.083024 32578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:59:51.089773 32578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:59:51.094867 32767 heartbeater.cc:503] Master 127.15.18.126:44147 was elected leader, sending a full tablet report...
I20260929 02:59:51.114481 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.101832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:36517 (local address 127.0.0.1:43790)
0929 02:59:51.103659 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:51.103676 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:51.103792 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:51.105103 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:51.105112 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:51.105143 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:51.105567 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.105580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.107935 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.107941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.110479 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.110499 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.113321 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.113329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.113567 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.113624 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 02:59:51.113740 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 02:59:51.113963 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1646,"thread_start_us":293,"threads_started":1}
I20260929 02:59:51.115085 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.102403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.15.18.65:36517)
0929 02:59:51.104413 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.104421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.104452 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:59:51.104557 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.104566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.104839 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.105788 (+ 949us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:51.106055 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.107760 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.110732 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.112425 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.114162 (+ 1737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.114434 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.114506 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.114696 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1820,"thread_start_us":1632,"threads_started":1}
I20260929 02:59:51.119092 32702 tablet_service.cc:1467] Processing CreateTablet for tablet 757e177e67d444698b32f17d366451a1 (DEFAULT_TABLE table=test-table [id=d109a294a2734001a1e1b06711a16973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:51.120410 32702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757e177e67d444698b32f17d366451a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:51.121229 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.101804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:43913 (local address 127.0.0.1:52278)
0929 02:59:51.102745 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:51.102783 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:51.102960 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:51.113868 (+ 10908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:51.113873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:51.113898 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:51.114312 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.114441 (+ 129us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.117020 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.117025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.118649 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.118662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.120498 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.120505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.120657 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.120689 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:59:51.120784 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:59:51.120916 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":1479,"threads_started":1}
I20260929 02:59:51.121320 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.101850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.15.18.66:43913)
0929 02:59:51.113247 (+ 11397us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.113256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.113285 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:59:51.113396 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.113405 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.113665 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.113862 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:51.114640 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.116905 (+ 2265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.118846 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.120282 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.120421 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.120603 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.120840 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.121006 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11255,"thread_start_us":11150,"threads_started":1}
I20260929 02:59:51.122815 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.103699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:36257 (local address 127.0.0.1:44902)
0929 02:59:51.112949 (+ 9250us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:51.112968 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:51.113088 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:51.116101 (+ 3013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:51.116106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:51.116132 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:51.116443 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.116454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.118259 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.118264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.120297 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.120311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.122074 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.122081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.122237 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.122268 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:59:51.122397 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 02:59:51.122519 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9116,"thread_start_us":8972,"threads_started":1}
I20260929 02:59:51.122907 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.104014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.15.18.67:36257)
0929 02:59:51.115608 (+ 11594us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.115616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.115651 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:59:51.115741 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.115745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.115923 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.116080 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:51.116620 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.118113 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.120462 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.121879 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.122000 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.122200 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.122440 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.122596 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11407,"thread_start_us":11243,"threads_started":1}
I20260929 02:59:51.124891 368 tablet_service.cc:1467] Processing CreateTablet for tablet 757e177e67d444698b32f17d366451a1 (DEFAULT_TABLE table=test-table [id=d109a294a2734001a1e1b06711a16973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:51.126106 368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757e177e67d444698b32f17d366451a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:51.126559 502 tablet_service.cc:1467] Processing CreateTablet for tablet 757e177e67d444698b32f17d366451a1 (DEFAULT_TABLE table=test-table [id=d109a294a2734001a1e1b06711a16973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:51.127954 502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757e177e67d444698b32f17d366451a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:51.135263 586 tablet_bootstrap.cc:492] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: Bootstrap starting.
I20260929 02:59:51.139228 586 tablet_bootstrap.cc:654] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:51.140226 587 tablet_bootstrap.cc:492] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: Bootstrap starting.
I20260929 02:59:51.141549 588 tablet_bootstrap.cc:492] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: Bootstrap starting.
I20260929 02:59:51.142333 586 log.cc:824] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:51.143796 587 tablet_bootstrap.cc:654] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:51.144711 587 log.cc:824] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:51.144716 588 tablet_bootstrap.cc:654] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:51.145606 586 tablet_bootstrap.cc:492] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: No bootstrap required, opened a new log
I20260929 02:59:51.145838 588 log.cc:824] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:51.145885 586 ts_tablet_manager.cc:1397] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260929 02:59:51.147323 587 tablet_bootstrap.cc:492] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: No bootstrap required, opened a new log
I20260929 02:59:51.147583 587 ts_tablet_manager.cc:1397] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 02:59:51.148669 588 tablet_bootstrap.cc:492] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: No bootstrap required, opened a new log
I20260929 02:59:51.148911 588 ts_tablet_manager.cc:1397] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:59:51.155014 587 raft_consensus.cc:348] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 } } peers { permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 } } peers { permanent_uuid: "40ce0e0bfef245ffb5b1748447101717" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 36257 } }
I20260929 02:59:51.155378 587 raft_consensus.cc:374] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:51.155493 587 raft_consensus.cc:729] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d280df9f7094275aec3fda15c8e3ade, State: Initialized, Role: FOLLOWER
I20260929 02:59:51.156088 587 consensus_queue.cc:260] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 } } peers { permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 } } peers { permanent_uuid: "40ce0e0bfef245ffb5b1748447101717" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 36257 } }
I20260929 02:59:51.155917 586 raft_consensus.cc:348] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 } } peers { permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 } } peers { permanent_uuid: "40ce0e0bfef245ffb5b1748447101717" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 36257 } }
I20260929 02:59:51.156371 586 raft_consensus.cc:374] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:51.156502 586 raft_consensus.cc:729] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5237f7dcc24a3db57a75c4a14d1ee5, State: Initialized, Role: FOLLOWER
I20260929 02:59:51.157055 586 consensus_queue.cc:260] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 } } peers { permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 } } peers { permanent_uuid: "40ce0e0bfef245ffb5b1748447101717" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 36257 } }
I20260929 02:59:51.158828 586 ts_tablet_manager.cc:1428] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 02:59:51.158840 588 raft_consensus.cc:348] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 } } peers { permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 } } peers { permanent_uuid: "40ce0e0bfef245ffb5b1748447101717" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 36257 } }
I20260929 02:59:51.159246 588 raft_consensus.cc:374] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:51.159382 588 raft_consensus.cc:729] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ce0e0bfef245ffb5b1748447101717, State: Initialized, Role: FOLLOWER
I20260929 02:59:51.159973 434 heartbeater.cc:503] Master 127.15.18.126:44147 was elected leader, sending a full tablet report...
I20260929 02:59:51.160063 588 consensus_queue.cc:260] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 } } peers { permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 } } peers { permanent_uuid: "40ce0e0bfef245ffb5b1748447101717" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 36257 } }
I20260929 02:59:51.160560 587 ts_tablet_manager.cc:1428] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260929 02:59:51.161412 568 heartbeater.cc:503] Master 127.15.18.126:44147 was elected leader, sending a full tablet report...
I20260929 02:59:51.162002 588 ts_tablet_manager.cc:1428] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260929 02:59:51.287827 569 tablet.cc:2369] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:59:51.293581 594 raft_consensus.cc:482] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:59:51.293859 594 raft_consensus.cc:504] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 } } peers { permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 } } peers { permanent_uuid: "40ce0e0bfef245ffb5b1748447101717" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 36257 } }
I20260929 02:59:51.295677 594 leader_election.cc:288] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d280df9f7094275aec3fda15c8e3ade (127.15.18.66:43913), dd5237f7dcc24a3db57a75c4a14d1ee5 (127.15.18.65:36517)
I20260929 02:59:51.302016 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.296291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:46815 (local address 127.15.18.65:36517)
0929 02:59:51.296481 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.296486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.296503 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:59:51.296561 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.296565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.296621 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.296695 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:51.297130 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.298091 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.300033 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.301185 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.301303 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.301358 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.301710 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.301831 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:59:51.302096 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.295934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:36517 (local address 127.15.18.67:46815)
0929 02:59:51.296216 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:51.296237 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:51.296365 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:51.296754 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:51.296759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:51.296794 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:51.296969 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.296979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.298245 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.298251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.299875 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.299890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.301336 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.301343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.301551 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.301588 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:59:51.301691 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 02:59:51.301826 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260929 02:59:51.303058 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.296178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:42305 (local address 127.15.18.66:43913)
0929 02:59:51.296408 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.296414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.297027 (+ 613us) server_negotiation.cc:407] Connection header received
0929 02:59:51.297173 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.297177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.297235 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.297298 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:51.297875 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.298884 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.301052 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.302218 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.302361 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.302430 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.302743 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.302871 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 02:59:51.303133 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.296171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:43913 (local address 127.15.18.67:42305)
0929 02:59:51.296926 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:51.296940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:51.297044 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:51.297350 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:51.297354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:51.297379 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:51.297710 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.297721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.299030 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.299034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.300863 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:51.300878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.302459 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.302466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:51.302580 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.302604 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:59:51.302679 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 02:59:51.302830 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":419,"threads_started":1}
I20260929 02:59:51.303287 32722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757e177e67d444698b32f17d366451a1" candidate_uuid: "40ce0e0bfef245ffb5b1748447101717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" is_pre_election: true
I20260929 02:59:51.303965 32722 raft_consensus.cc:2454] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40ce0e0bfef245ffb5b1748447101717 in term 0.
I20260929 02:59:51.304275 388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757e177e67d444698b32f17d366451a1" candidate_uuid: "40ce0e0bfef245ffb5b1748447101717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d280df9f7094275aec3fda15c8e3ade" is_pre_election: true
I20260929 02:59:51.304639 458 leader_election.cc:302] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40ce0e0bfef245ffb5b1748447101717, dd5237f7dcc24a3db57a75c4a14d1ee5; no voters:
I20260929 02:59:51.304875 388 raft_consensus.cc:2454] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40ce0e0bfef245ffb5b1748447101717 in term 0.
I20260929 02:59:51.305035 594 raft_consensus.cc:2781] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:59:51.305187 594 raft_consensus.cc:482] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:59:51.305297 594 raft_consensus.cc:3037] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:59:51.306753 435 tablet.cc:2369] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:59:51.307679 594 raft_consensus.cc:504] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 } } peers { permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 } } peers { permanent_uuid: "40ce0e0bfef245ffb5b1748447101717" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 36257 } }
I20260929 02:59:51.308306 594 leader_election.cc:288] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [CANDIDATE]: Term 1 election: Requested vote from peers 6d280df9f7094275aec3fda15c8e3ade (127.15.18.66:43913), dd5237f7dcc24a3db57a75c4a14d1ee5 (127.15.18.65:36517)
I20260929 02:59:51.308773 388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757e177e67d444698b32f17d366451a1" candidate_uuid: "40ce0e0bfef245ffb5b1748447101717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d280df9f7094275aec3fda15c8e3ade"
I20260929 02:59:51.308959 388 raft_consensus.cc:3037] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:51.308938 32722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757e177e67d444698b32f17d366451a1" candidate_uuid: "40ce0e0bfef245ffb5b1748447101717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5"
I20260929 02:59:51.309137 32722 raft_consensus.cc:3037] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:51.311404 388 raft_consensus.cc:2454] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40ce0e0bfef245ffb5b1748447101717 in term 1.
I20260929 02:59:51.311995 457 leader_election.cc:302] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40ce0e0bfef245ffb5b1748447101717, 6d280df9f7094275aec3fda15c8e3ade; no voters:
I20260929 02:59:51.312081 32722 raft_consensus.cc:2454] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40ce0e0bfef245ffb5b1748447101717 in term 1.
I20260929 02:59:51.313247 594 raft_consensus.cc:2781] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:59:51.313735 594 raft_consensus.cc:686] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 1 LEADER]: Becoming Leader. State: Replica: 40ce0e0bfef245ffb5b1748447101717, State: Running, Role: LEADER
I20260929 02:59:51.314256 594 consensus_queue.cc:237] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 } } peers { permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 } } peers { permanent_uuid: "40ce0e0bfef245ffb5b1748447101717" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 36257 } }
W20260929 02:59:51.316131 300 tablet.cc:2369] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:59:51.318928 32578 catalog_manager.cc:5495] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40ce0e0bfef245ffb5b1748447101717 (127.15.18.67). New cstate: current_term: 1 leader_uuid: "40ce0e0bfef245ffb5b1748447101717" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40ce0e0bfef245ffb5b1748447101717" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 36257 } health_report { overall_health: HEALTHY } } }
I20260929 02:59:51.351878 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.346166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.15.18.67:36257)
0929 02:59:51.346429 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.346435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.346455 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:59:51.346603 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.346607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.346684 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.346813 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:59:51.347208 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.348030 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.349550 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.349952 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.350062 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.350536 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:59:51.351223 (+ 687us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:59:51.351398 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.351499 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.351612 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:59:51.359934 32722 raft_consensus.cc:1261] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5 [term 1 FOLLOWER]: Refusing update from remote peer 40ce0e0bfef245ffb5b1748447101717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:59:51.359936 388 raft_consensus.cc:1261] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade [term 1 FOLLOWER]: Refusing update from remote peer 40ce0e0bfef245ffb5b1748447101717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:59:51.360813 594 consensus_queue.cc:1035] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5237f7dcc24a3db57a75c4a14d1ee5" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:59:51.361090 596 consensus_queue.cc:1035] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d280df9f7094275aec3fda15c8e3ade" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:59:51.387938 32577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:59:51.388208 32577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:59:51.391781 32577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae0598cf20b0414fa2d6ce57d8ac773d: Sending DeleteTablet for 3 replicas of tablet 757e177e67d444698b32f17d366451a1
I20260929 02:59:51.393008 502 tablet_service.cc:1514] Processing DeleteTablet for tablet 757e177e67d444698b32f17d366451a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:51 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:44902
I20260929 02:59:51.393003 32702 tablet_service.cc:1514] Processing DeleteTablet for tablet 757e177e67d444698b32f17d366451a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:51 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:43790
I20260929 02:59:51.393111 368 tablet_service.cc:1514] Processing DeleteTablet for tablet 757e177e67d444698b32f17d366451a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:51 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:52278
I20260929 02:59:51.395375 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 32639
I20260929 02:59:51.403728 610 tablet_replica.cc:330] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: stopping tablet replica
I20260929 02:59:51.404529 610 raft_consensus.cc:2229] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:59:51.405242 610 raft_consensus.cc:2258] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:51.406472 608 tablet_replica.cc:330] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: stopping tablet replica
I20260929 02:59:51.407006 608 raft_consensus.cc:2229] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:59:51.407785 610 ts_tablet_manager.cc:1905] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:59:51.407835 608 raft_consensus.cc:2258] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:51.410179 608 ts_tablet_manager.cc:1905] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:59:51.410796 609 tablet_replica.cc:330] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: stopping tablet replica
I20260929 02:59:51.411223 609 raft_consensus.cc:2229] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:59:51.411746 609 raft_consensus.cc:2258] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:51.414664 609 ts_tablet_manager.cc:1905] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:59:51.419004 608 ts_tablet_manager.cc:1918] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:59:51.419231 608 log.cc:1196] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/757e177e67d444698b32f17d366451a1
I20260929 02:59:51.419822 608 ts_tablet_manager.cc:1939] T 757e177e67d444698b32f17d366451a1 P 40ce0e0bfef245ffb5b1748447101717: Deleting consensus metadata
I20260929 02:59:51.421159 610 ts_tablet_manager.cc:1918] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:59:51.421770 32564 catalog_manager.cc:4841] TS 40ce0e0bfef245ffb5b1748447101717 (127.15.18.67:36257): tablet 757e177e67d444698b32f17d366451a1 (table test-table [id=d109a294a2734001a1e1b06711a16973]) successfully deleted
I20260929 02:59:51.421978 609 ts_tablet_manager.cc:1918] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:59:51.422152 609 log.cc:1196] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/757e177e67d444698b32f17d366451a1
I20260929 02:59:51.422681 609 ts_tablet_manager.cc:1939] T 757e177e67d444698b32f17d366451a1 P dd5237f7dcc24a3db57a75c4a14d1ee5: Deleting consensus metadata
I20260929 02:59:51.424465 610 log.cc:1196] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/757e177e67d444698b32f17d366451a1
Sep 29 02:59:51 dist-test-slave-q5f0 krb5kdc[32533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:59:51.425051 610 ts_tablet_manager.cc:1939] T 757e177e67d444698b32f17d366451a1 P 6d280df9f7094275aec3fda15c8e3ade: Deleting consensus metadata
I20260929 02:59:51.426796 32564 catalog_manager.cc:4841] TS 6d280df9f7094275aec3fda15c8e3ade (127.15.18.66:43913): tablet 757e177e67d444698b32f17d366451a1 (table test-table [id=d109a294a2734001a1e1b06711a16973]) successfully deleted
I20260929 02:59:51.427488 32565 catalog_manager.cc:4841] TS dd5237f7dcc24a3db57a75c4a14d1ee5 (127.15.18.65:36517): tablet 757e177e67d444698b32f17d366451a1 (table test-table [id=d109a294a2734001a1e1b06711a16973]) successfully deleted
I20260929 02:59:51.436093 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.397154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.15.18.65:36517)
0929 02:59:51.397339 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.397345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.401642 (+ 4297us) server_negotiation.cc:407] Connection header received
0929 02:59:51.401839 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.401845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.401941 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.402062 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:51.413776 (+ 11714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.414948 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.417043 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.417677 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.417798 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.426370 (+ 8572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:51.426385 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:51.426403 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:51.426453 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:51.433300 (+ 6847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:51.434047 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:51.434055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:51.434057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:51.434190 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:51.434571 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:51.434574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:51.434576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:51.434930 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:51.435196 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.435548 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.435759 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:59:51.436678 32762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:51.563159 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 32639
I20260929 02:59:51.583923 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 305
Sep 29 02:59:51 dist-test-slave-q5f0 krb5kdc[32533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:59:51.600960 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.584730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.15.18.66:43913)
0929 02:59:51.584954 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.584960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.585045 (+ 85us) server_negotiation.cc:407] Connection header received
0929 02:59:51.585246 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.585252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.585343 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.585427 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:51.586663 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.587426 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.588768 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.589169 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.589287 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.593758 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:51.593781 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:51.593800 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:51.593850 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:51.597247 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:51.598134 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:51.598144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:51.598146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:51.598304 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:51.598796 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:51.598803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:51.598806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:51.599270 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:51.599699 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.600342 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.600584 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:59:51.601697 429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:51.692842 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 305
I20260929 02:59:51.713819 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 439
Sep 29 02:59:51 dist-test-slave-q5f0 krb5kdc[32533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:59:51.732806 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:51.714618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.15.18.67:36257)
0929 02:59:51.714828 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:51.714834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:51.714866 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:59:51.714950 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:51.714956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:51.715063 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:51.715186 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:51.716601 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.717835 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.719176 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:51.719887 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:51.720032 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:51.724758 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:51.724779 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:51.724800 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:51.724869 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:51.729097 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:51.730091 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:51.730102 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:51.730106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:51.730329 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:51.730834 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:51.730839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:51.730843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:51.731365 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:51.731950 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:51.732125 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 02:59:51.732364 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 02:59:51.733680 563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:51.857076 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 439
I20260929 02:59:51.872822 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 32546
I20260929 02:59:51.873978 32608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:51.984302 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 32546
2026-09-29T02:59:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4213 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T02:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:59:52Z Disabled control of system clock
I20260929 02:59:52.054582 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43595
--webserver_interface=127.15.18.126
--webserver_port=0
--builtin_ntp_servers=127.15.18.84:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:59:52.386868 619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:52.387151 619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:52.387233 619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:52.394451 619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:59:52.394564 619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:52.394600 619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:52.394652 619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:59:52.394695 619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:59:52.403983 619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:52.405261 619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:52.407115 619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:52.418540 625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:52.418594 627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:52.418645 624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:52.419108 619 server_base.cc:1034] running on GCE node
I20260929 02:59:52.419895 619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:52.421269 619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:52.422506 619 hybrid_clock.cc:648] HybridClock initialized: now 1790650792422444 us; error 52 us; skew 500 ppm
I20260929 02:59:52.422891 619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:59:52.425216 619 webserver.cc:460] Webserver started at http://127.15.18.126:38373/ using document root <none> and password file <none>
I20260929 02:59:52.425907 619 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:52.426002 619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:52.426304 619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:52.429265 619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "9a16cd981e054ffeba202d620a7a3a20"
format_stamp: "Formatted at 2026-09-29 02:59:52 on dist-test-slave-q5f0"
server_key: "677edb54726bac2d8c392afecda0dd08"
server_key_iv: "09a124c9782be20163670d01509b8ba7"
server_key_version: "encryptionkey@0"
I20260929 02:59:52.430100 619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "9a16cd981e054ffeba202d620a7a3a20"
format_stamp: "Formatted at 2026-09-29 02:59:52 on dist-test-slave-q5f0"
server_key: "677edb54726bac2d8c392afecda0dd08"
server_key_iv: "09a124c9782be20163670d01509b8ba7"
server_key_version: "encryptionkey@0"
I20260929 02:59:52.435767 619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:59:52.439582 633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:52.441134 619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:59:52.441318 619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "9a16cd981e054ffeba202d620a7a3a20"
format_stamp: "Formatted at 2026-09-29 02:59:52 on dist-test-slave-q5f0"
server_key: "677edb54726bac2d8c392afecda0dd08"
server_key_iv: "09a124c9782be20163670d01509b8ba7"
server_key_version: "encryptionkey@0"
I20260929 02:59:52.441550 619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:52.455142 619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:52.456030 619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:52.456307 619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:52.473801 619 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:43595
I20260929 02:59:52.473821 685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:43595 every 8 connection(s)
I20260929 02:59:52.475148 619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:59:52.478137 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 619
I20260929 02:59:52.478307 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:59:52.478843 15433 external_mini_cluster.cc:1428] Setting key 4d54f17e58418607a61300d4e78af722
I20260929 02:59:52.479259 686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:52.493125 686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20: Bootstrap starting.
I20260929 02:59:52.493136 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:52.480800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.15.18.126:43595)
0929 02:59:52.481747 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:52.481758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:52.481803 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:59:52.483911 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:52.483938 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:52.484429 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:52.484769 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:52.485378 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:52.486741 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:52.488649 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:52.489365 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:52.489523 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:52.490791 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:52.490799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:52.490815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:59:52.491386 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:52.491464 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:59:52.491488 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:52.491611 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:52.491864 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 02:59:52.492183 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":340,"threads_started":1}
I20260929 02:59:52.496240 686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:52.497184 686 log.cc:824] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:52.500036 686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20: No bootstrap required, opened a new log
I20260929 02:59:52.506989 686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16cd981e054ffeba202d620a7a3a20" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43595 } }
I20260929 02:59:52.507352 686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:52.507447 686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a16cd981e054ffeba202d620a7a3a20, State: Initialized, Role: FOLLOWER
I20260929 02:59:52.508045 686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16cd981e054ffeba202d620a7a3a20" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43595 } }
I20260929 02:59:52.508258 686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:59:52.508383 686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:59:52.508530 686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:52.510633 686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16cd981e054ffeba202d620a7a3a20" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43595 } }
I20260929 02:59:52.511135 686 leader_election.cc:302] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a16cd981e054ffeba202d620a7a3a20; no voters:
I20260929 02:59:52.511703 686 leader_election.cc:288] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:59:52.511822 691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:59:52.512463 691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [term 1 LEADER]: Becoming Leader. State: Replica: 9a16cd981e054ffeba202d620a7a3a20, State: Running, Role: LEADER
I20260929 02:59:52.512959 691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16cd981e054ffeba202d620a7a3a20" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43595 } }
I20260929 02:59:52.513731 686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:59:52.515448 692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a16cd981e054ffeba202d620a7a3a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16cd981e054ffeba202d620a7a3a20" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43595 } } }
I20260929 02:59:52.515903 692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:52.517747 693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a16cd981e054ffeba202d620a7a3a20. Latest consensus state: current_term: 1 leader_uuid: "9a16cd981e054ffeba202d620a7a3a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a16cd981e054ffeba202d620a7a3a20" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43595 } } }
I20260929 02:59:52.518049 693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20 [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:52.519402 698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:59:52.524061 698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:59:52.533035 698 catalog_manager.cc:1347] Generated new cluster ID: c89cbda15d4a477886478de470c629d6
I20260929 02:59:52.533162 698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:59:52.567734 698 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:59:52.569044 698 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:59:52.584878 698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20: Generated new TSK 0
I20260929 02:59:52.585768 698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:59:52.598984 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43595
--builtin_ntp_servers=127.15.18.84:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:59:52.941953 710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:52.942232 710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:52.942342 710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:52.949719 710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:52.949826 710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:52.949973 710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:59:52.959409 710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:52.960827 710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:52.962599 710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:52.975456 715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:52.975409 718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:52.975479 716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:52.976307 710 server_base.cc:1034] running on GCE node
I20260929 02:59:52.976881 710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:52.977797 710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:52.979027 710 hybrid_clock.cc:648] HybridClock initialized: now 1790650792978957 us; error 57 us; skew 500 ppm
I20260929 02:59:52.979323 710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:59:52.981564 710 webserver.cc:460] Webserver started at http://127.15.18.65:39319/ using document root <none> and password file <none>
I20260929 02:59:52.982225 710 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:52.982311 710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:52.982590 710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:52.985517 710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "c192c27ab65c48db820124d75ad0956e"
format_stamp: "Formatted at 2026-09-29 02:59:52 on dist-test-slave-q5f0"
server_key: "8f80a7581a2abc50623fee2dc593911a"
server_key_iv: "844a011a13f79e8a1aee3b235b1683c8"
server_key_version: "encryptionkey@0"
I20260929 02:59:52.986279 710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "c192c27ab65c48db820124d75ad0956e"
format_stamp: "Formatted at 2026-09-29 02:59:52 on dist-test-slave-q5f0"
server_key: "8f80a7581a2abc50623fee2dc593911a"
server_key_iv: "844a011a13f79e8a1aee3b235b1683c8"
server_key_version: "encryptionkey@0"
I20260929 02:59:52.991813 710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:59:52.995726 724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:52.997368 710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:59:52.997531 710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "c192c27ab65c48db820124d75ad0956e"
format_stamp: "Formatted at 2026-09-29 02:59:52 on dist-test-slave-q5f0"
server_key: "8f80a7581a2abc50623fee2dc593911a"
server_key_iv: "844a011a13f79e8a1aee3b235b1683c8"
server_key_version: "encryptionkey@0"
I20260929 02:59:52.997751 710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:53.010521 710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:53.011389 710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:53.011724 710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:53.012670 710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:53.014184 710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:53.014272 710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:53.014375 710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:53.014446 710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:53.045256 710 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:44979
I20260929 02:59:53.045333 837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:44979 every 8 connection(s)
I20260929 02:59:53.046635 710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:59:53.049655 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 710
I20260929 02:59:53.049840 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:59:53.050429 15433 external_mini_cluster.cc:1428] Setting key a5aa8d723000967a4815c407efb9bb30
I20260929 02:59:53.053236 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43595
--builtin_ntp_servers=127.15.18.84:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:59:53.058916 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.049011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:37977 (local address 127.15.18.126:43595)
0929 02:59:53.049445 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:53.049450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:53.050780 (+ 1330us) server_negotiation.cc:407] Connection header received
0929 02:59:53.052714 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:53.052719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:53.052800 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:53.052898 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:53.053997 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.054711 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.056311 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.056711 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.056818 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:53.058101 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:53.058105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:53.058108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:59:53.058126 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:53.058165 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:59:53.058177 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:53.058248 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:53.058470 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 02:59:53.058566 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":210,"threads_started":1}
I20260929 02:59:53.060213 841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.049657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43595 (local address 127.15.18.65:37977)
0929 02:59:53.050634 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:53.050675 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:53.052474 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:53.053118 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:53.053126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:53.053221 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:53.053840 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:53.053859 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.054850 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.054855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:53.056181 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:53.056194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.056915 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.056922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:53.057141 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:53.057801 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:59:53.057803 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:53.057846 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:59:53.057923 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:53.058258 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:53.058269 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:53.058278 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:59:53.058454 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 02:59:53.058845 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":286,"threads_started":1}
I20260929 02:59:53.061741 838 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43595
I20260929 02:59:53.062018 838 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:53.062773 838 heartbeater.cc:511] Master 127.15.18.126:43595 requested a full tablet report, sending...
I20260929 02:59:53.064885 651 ts_manager.cc:194] Registered new tserver with Master: c192c27ab65c48db820124d75ad0956e (127.15.18.65:44979)
I20260929 02:59:53.066407 651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.65:37977
W20260929 02:59:53.377754 842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:53.378063 842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:53.378191 842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:53.385746 842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:53.385859 842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:53.385994 842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:59:53.395411 842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:53.396803 842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:53.398593 842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:53.411598 850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:53.411649 847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:53.411628 848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:53.412236 842 server_base.cc:1034] running on GCE node
I20260929 02:59:53.412871 842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:53.413831 842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:53.415045 842 hybrid_clock.cc:648] HybridClock initialized: now 1790650793414979 us; error 57 us; skew 500 ppm
I20260929 02:59:53.415402 842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:59:53.417749 842 webserver.cc:460] Webserver started at http://127.15.18.66:45819/ using document root <none> and password file <none>
I20260929 02:59:53.418417 842 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:53.418510 842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:53.418834 842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:53.421720 842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "6b408a99195b4ef7a317186ba03592c6"
format_stamp: "Formatted at 2026-09-29 02:59:53 on dist-test-slave-q5f0"
server_key: "7604561f797ca24341f5fa398d882a76"
server_key_iv: "fdb1d3f4962cc7003b2cd60733d3fe0b"
server_key_version: "encryptionkey@0"
I20260929 02:59:53.422482 842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "6b408a99195b4ef7a317186ba03592c6"
format_stamp: "Formatted at 2026-09-29 02:59:53 on dist-test-slave-q5f0"
server_key: "7604561f797ca24341f5fa398d882a76"
server_key_iv: "fdb1d3f4962cc7003b2cd60733d3fe0b"
server_key_version: "encryptionkey@0"
I20260929 02:59:53.428097 842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:59:53.431907 856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:53.433256 842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:59:53.433423 842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "6b408a99195b4ef7a317186ba03592c6"
format_stamp: "Formatted at 2026-09-29 02:59:53 on dist-test-slave-q5f0"
server_key: "7604561f797ca24341f5fa398d882a76"
server_key_iv: "fdb1d3f4962cc7003b2cd60733d3fe0b"
server_key_version: "encryptionkey@0"
I20260929 02:59:53.433624 842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:53.450825 842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:53.451730 842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:53.452018 842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:53.453034 842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:53.454569 842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:53.454674 842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:53.454783 842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:53.454870 842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:53.485507 842 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:43993
I20260929 02:59:53.485584 969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:43993 every 8 connection(s)
I20260929 02:59:53.486882 842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:59:53.497254 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 842
I20260929 02:59:53.497406 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:59:53.497865 15433 external_mini_cluster.cc:1428] Setting key 5c2e7c35535688696bdfd013a7a2005c
I20260929 02:59:53.499270 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.489593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:38873 (local address 127.15.18.126:43595)
0929 02:59:53.489804 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:53.489810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:53.491338 (+ 1528us) server_negotiation.cc:407] Connection header received
0929 02:59:53.493142 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:53.493147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:53.493214 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:53.493303 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:53.494317 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.495044 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.496543 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.496947 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.497072 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:53.498490 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:53.498495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:53.498499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:59:53.498516 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:53.498549 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:59:53.498560 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:53.498636 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:53.498857 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 02:59:53.498957 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:59:53.500635 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.490123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43595 (local address 127.15.18.66:38873)
0929 02:59:53.491154 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:53.491208 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:53.492887 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:53.493485 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:53.493493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:53.493581 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:53.494188 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:53.494206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.495168 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.495172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:53.496431 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:53.496442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.497165 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.497170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:53.497401 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:53.498174 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:59:53.498176 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:53.498228 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:59:53.498308 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:53.498657 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:53.498668 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:53.498675 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:59:53.498829 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 02:59:53.499248 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":296,"threads_started":1}
I20260929 02:59:53.500946 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43595
--builtin_ntp_servers=127.15.18.84:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:59:53.502162 970 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43595
I20260929 02:59:53.502482 970 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:53.503340 970 heartbeater.cc:511] Master 127.15.18.126:43595 requested a full tablet report, sending...
I20260929 02:59:53.504781 651 ts_manager.cc:194] Registered new tserver with Master: 6b408a99195b4ef7a317186ba03592c6 (127.15.18.66:43993)
I20260929 02:59:53.505582 651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.66:38873
W20260929 02:59:53.821969 973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:53.822254 973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:53.822368 973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:53.829581 973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:53.829682 973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:53.829828 973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:59:53.839017 973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:43595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:53.840368 973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:53.842108 973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:53.855185 980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:53.855185 979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:53.855134 982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:53.856395 973 server_base.cc:1034] running on GCE node
I20260929 02:59:53.857012 973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:53.857978 973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:53.859203 973 hybrid_clock.cc:648] HybridClock initialized: now 1790650793859140 us; error 52 us; skew 500 ppm
I20260929 02:59:53.859567 973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:59:53.861939 973 webserver.cc:460] Webserver started at http://127.15.18.67:43923/ using document root <none> and password file <none>
I20260929 02:59:53.862625 973 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:53.862713 973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:53.863021 973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:53.865939 973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "61c7670915d34d478c0079976d48bb66"
format_stamp: "Formatted at 2026-09-29 02:59:53 on dist-test-slave-q5f0"
server_key: "e8f57806287222c3eab95ff355ffd8eb"
server_key_iv: "e3b9f99e95ee5eb766cc9a78ef694711"
server_key_version: "encryptionkey@0"
I20260929 02:59:53.866676 973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "61c7670915d34d478c0079976d48bb66"
format_stamp: "Formatted at 2026-09-29 02:59:53 on dist-test-slave-q5f0"
server_key: "e8f57806287222c3eab95ff355ffd8eb"
server_key_iv: "e3b9f99e95ee5eb766cc9a78ef694711"
server_key_version: "encryptionkey@0"
I20260929 02:59:53.871752 973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:59:53.875334 988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:53.876796 973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:59:53.876937 973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "61c7670915d34d478c0079976d48bb66"
format_stamp: "Formatted at 2026-09-29 02:59:53 on dist-test-slave-q5f0"
server_key: "e8f57806287222c3eab95ff355ffd8eb"
server_key_iv: "e3b9f99e95ee5eb766cc9a78ef694711"
server_key_version: "encryptionkey@0"
I20260929 02:59:53.877151 973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:53.900719 973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:53.901588 973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:53.901949 973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:53.902992 973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:53.904461 973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:53.904546 973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:53.904645 973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:53.904719 973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:53.933590 973 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:44969
I20260929 02:59:53.933647 1100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:44969 every 8 connection(s)
I20260929 02:59:53.934860 973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:59:53.944482 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 973
I20260929 02:59:53.944649 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:59:53.945150 15433 external_mini_cluster.cc:1428] Setting key c2df522c025808e9c09375d97fd5f2c1
I20260929 02:59:53.947214 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.937442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:59113 (local address 127.15.18.126:43595)
0929 02:59:53.937689 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:53.937695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:53.939030 (+ 1335us) server_negotiation.cc:407] Connection header received
0929 02:59:53.940827 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:53.940833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:53.940905 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:53.940998 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:53.942033 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.942804 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.944359 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.944809 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.944950 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:53.946383 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:53.946390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:53.946395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:59:53.946415 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:53.946447 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:59:53.946460 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:53.946527 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:53.946806 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 02:59:53.946906 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:59:53.948310 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.938016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43595 (local address 127.15.18.67:59113)
0929 02:59:53.938891 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:53.938931 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:53.940562 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:53.941185 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:53.941193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:53.941285 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:53.941863 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:53.941879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.942953 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.942957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:53.944216 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:53.944228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.945072 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.945080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:53.945311 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:53.946064 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:59:53.946066 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:53.946105 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:59:53.946197 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:53.946551 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:53.946561 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:53.946568 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:59:53.946736 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 02:59:53.947123 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":266,"threads_started":1}
I20260929 02:59:53.949668 1101 heartbeater.cc:347] Connected to a master server at 127.15.18.126:43595
I20260929 02:59:53.949965 1101 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:53.950660 1101 heartbeater.cc:511] Master 127.15.18.126:43595 requested a full tablet report, sending...
I20260929 02:59:53.952019 651 ts_manager.cc:194] Registered new tserver with Master: 61c7670915d34d478c0079976d48bb66 (127.15.18.67:44969)
I20260929 02:59:53.952792 651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.67:59113
I20260929 02:59:53.960713 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:59:53.970778 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.964963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.15.18.126:43595)
0929 02:59:53.965172 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:53.965178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:53.965510 (+ 332us) server_negotiation.cc:407] Connection header received
0929 02:59:53.965697 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:53.965703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:53.965791 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:53.965877 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:53.966413 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.967118 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.968544 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.968970 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.969078 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:53.970125 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:53.970130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:53.970133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:59:53.970151 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:53.970183 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:59:53.970194 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:53.970273 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:53.970394 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 02:59:53.970489 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:59:53.974438 651 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:59:53.981262 651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:59:54.003947 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.993668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.15.18.66:43993)
0929 02:59:53.994212 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:53.994226 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:53.994254 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:59:53.994517 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:53.994525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:53.994794 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:53.994994 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:53.995965 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.997825 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.001018 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.002663 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.002794 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.003081 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:54.003435 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 02:59:54.003636 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":286,"threads_started":1}
I20260929 02:59:54.004981 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.993420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:43993 (local address 127.0.0.1:42216)
0929 02:59:53.994101 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:53.994144 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:53.994304 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:53.995280 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:53.995289 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:53.995319 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:53.995757 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:53.995773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:53.998048 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:53.998055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:54.000697 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:54.000723 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.002916 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.002922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:54.003147 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.003206 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 02:59:54.004332 (+ 1126us) client_negotiation.cc:240] Negotiation successful
0929 02:59:54.004580 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":271,"threads_started":1}
I20260929 02:59:54.009307 904 tablet_service.cc:1467] Processing CreateTablet for tablet 3e47f2822a70418092991cae5e011c12 (DEFAULT_TABLE table=test-table [id=7e5072e631ca45a4b3f01f9e64c13f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:54.010123 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.994472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:44969 (local address 127.0.0.1:46594)
0929 02:59:53.995003 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:53.995019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:53.995119 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:54.003220 (+ 8101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:54.003224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:54.003265 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:54.003689 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:54.003703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.005886 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.005892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:54.008061 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:54.008074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.009399 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.009422 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:54.009596 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.009628 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:59:54.009708 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 02:59:54.009838 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
I20260929 02:59:54.010669 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.994814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.15.18.67:44969)
0929 02:59:54.001776 (+ 6962us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:54.001786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:54.001827 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:59:54.001935 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:54.001944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:54.002271 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:54.003245 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:54.003911 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.005397 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.008259 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.009195 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.009345 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.010047 (+ 702us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:54.010126 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 02:59:54.010313 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6753,"thread_start_us":6982,"threads_started":1}
I20260929 02:59:54.011180 904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e47f2822a70418092991cae5e011c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:54.011435 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.995032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44979 (local address 127.0.0.1:34798)
0929 02:59:54.002470 (+ 7438us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:54.002490 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:54.002624 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:54.004699 (+ 2075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:54.004704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:54.004734 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:54.005076 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:54.005088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.006851 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.006856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:54.008951 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:54.008966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.010714 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.010720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:54.010890 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.010921 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:59:54.011009 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:59:54.011133 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7308,"thread_start_us":7161,"threads_started":1}
I20260929 02:59:54.011605 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:53.995550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.15.18.65:44979)
0929 02:59:54.003945 (+ 8395us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:54.003953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:54.003983 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:59:54.004087 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:54.004094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:54.004450 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:54.004674 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:54.005295 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.006720 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.009167 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.010521 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.010655 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.010882 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:54.011086 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 02:59:54.011254 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8220,"thread_start_us":8277,"threads_started":1}
I20260929 02:59:54.015441 772 tablet_service.cc:1467] Processing CreateTablet for tablet 3e47f2822a70418092991cae5e011c12 (DEFAULT_TABLE table=test-table [id=7e5072e631ca45a4b3f01f9e64c13f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:54.015441 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 3e47f2822a70418092991cae5e011c12 (DEFAULT_TABLE table=test-table [id=7e5072e631ca45a4b3f01f9e64c13f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:54.017061 772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e47f2822a70418092991cae5e011c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:54.017441 1036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e47f2822a70418092991cae5e011c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:54.025542 1120 tablet_bootstrap.cc:492] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: Bootstrap starting.
I20260929 02:59:54.028644 1120 tablet_bootstrap.cc:654] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:54.029726 1120 log.cc:824] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:54.032624 1122 tablet_bootstrap.cc:492] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e: Bootstrap starting.
I20260929 02:59:54.033818 1120 tablet_bootstrap.cc:492] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: No bootstrap required, opened a new log
I20260929 02:59:54.034125 1120 ts_tablet_manager.cc:1397] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 02:59:54.036470 1122 tablet_bootstrap.cc:654] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:54.036659 1123 tablet_bootstrap.cc:492] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: Bootstrap starting.
I20260929 02:59:54.037901 1122 log.cc:824] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:54.040362 1123 tablet_bootstrap.cc:654] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:54.041332 1123 log.cc:824] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:54.042121 1122 tablet_bootstrap.cc:492] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e: No bootstrap required, opened a new log
I20260929 02:59:54.042416 1122 ts_tablet_manager.cc:1397] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 02:59:54.043803 1123 tablet_bootstrap.cc:492] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: No bootstrap required, opened a new log
I20260929 02:59:54.044031 1123 ts_tablet_manager.cc:1397] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260929 02:59:54.044191 1120 raft_consensus.cc:348] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 } } peers { permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 } } peers { permanent_uuid: "c192c27ab65c48db820124d75ad0956e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44979 } }
I20260929 02:59:54.044767 1120 raft_consensus.cc:374] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:54.044921 1120 raft_consensus.cc:729] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b408a99195b4ef7a317186ba03592c6, State: Initialized, Role: FOLLOWER
I20260929 02:59:54.045810 1120 consensus_queue.cc:260] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 } } peers { permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 } } peers { permanent_uuid: "c192c27ab65c48db820124d75ad0956e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44979 } }
I20260929 02:59:54.047806 1120 ts_tablet_manager.cc:1428] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 02:59:54.052685 970 heartbeater.cc:503] Master 127.15.18.126:43595 was elected leader, sending a full tablet report...
I20260929 02:59:54.052832 1123 raft_consensus.cc:348] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 } } peers { permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 } } peers { permanent_uuid: "c192c27ab65c48db820124d75ad0956e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44979 } }
I20260929 02:59:54.053277 1123 raft_consensus.cc:374] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:54.053362 1123 raft_consensus.cc:729] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c7670915d34d478c0079976d48bb66, State: Initialized, Role: FOLLOWER
I20260929 02:59:54.054028 1122 raft_consensus.cc:348] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 } } peers { permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 } } peers { permanent_uuid: "c192c27ab65c48db820124d75ad0956e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44979 } }
I20260929 02:59:54.054311 1123 consensus_queue.cc:260] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 } } peers { permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 } } peers { permanent_uuid: "c192c27ab65c48db820124d75ad0956e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44979 } }
I20260929 02:59:54.054517 1122 raft_consensus.cc:374] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:54.054630 1122 raft_consensus.cc:729] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c192c27ab65c48db820124d75ad0956e, State: Initialized, Role: FOLLOWER
I20260929 02:59:54.055361 1122 consensus_queue.cc:260] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 } } peers { permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 } } peers { permanent_uuid: "c192c27ab65c48db820124d75ad0956e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44979 } }
I20260929 02:59:54.062690 1101 heartbeater.cc:503] Master 127.15.18.126:43595 was elected leader, sending a full tablet report...
I20260929 02:59:54.063215 838 heartbeater.cc:503] Master 127.15.18.126:43595 was elected leader, sending a full tablet report...
I20260929 02:59:54.063486 1123 ts_tablet_manager.cc:1428] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20260929 02:59:54.064235 1122 ts_tablet_manager.cc:1428] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e: Time spent starting tablet: real 0.022s user 0.014s sys 0.005s
I20260929 02:59:54.134913 1128 raft_consensus.cc:482] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:59:54.135210 1128 raft_consensus.cc:504] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 } } peers { permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 } } peers { permanent_uuid: "c192c27ab65c48db820124d75ad0956e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44979 } }
I20260929 02:59:54.136940 1128 leader_election.cc:288] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b408a99195b4ef7a317186ba03592c6 (127.15.18.66:43993), 61c7670915d34d478c0079976d48bb66 (127.15.18.67:44969)
I20260929 02:59:54.143777 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.137261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:33313 (local address 127.15.18.66:43993)
0929 02:59:54.137434 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:54.137438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:54.137903 (+ 465us) server_negotiation.cc:407] Connection header received
0929 02:59:54.138132 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:54.138138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:54.138218 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:54.138317 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:54.138903 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.139885 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.141873 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.142827 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.142938 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.142988 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:54.143349 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 02:59:54.143458 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:59:54.143800 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.137211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:43993 (local address 127.15.18.65:33313)
0929 02:59:54.137811 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:54.137829 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:54.137969 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:54.138402 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:54.138406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:54.138437 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:54.138749 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:54.138763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.140011 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.140016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:54.141683 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:54.141695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.142997 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.143004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:54.143178 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.143214 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:59:54.143314 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:59:54.143455 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":36,"thread_start_us":305,"threads_started":1}
I20260929 02:59:54.144232 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.138068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:56223 (local address 127.15.18.67:44969)
0929 02:59:54.138289 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:54.138295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:54.138721 (+ 426us) server_negotiation.cc:407] Connection header received
0929 02:59:54.138865 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:54.138870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:54.138937 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:54.138999 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:54.139586 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.140502 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.142413 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.143346 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.143530 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.143598 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:54.143900 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:59:54.144025 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:59:54.144286 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.137954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:44969 (local address 127.15.18.65:56223)
0929 02:59:54.138620 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:54.138633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:54.138743 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:54.139038 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:54.139043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:54.139067 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:54.139422 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:54.139433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.140640 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.140643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:54.142183 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:54.142197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.143544 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.143551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:54.143719 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.143751 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:59:54.143849 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:59:54.143979 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":2,"thread_start_us":356,"threads_started":1}
I20260929 02:59:54.144995 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e47f2822a70418092991cae5e011c12" candidate_uuid: "c192c27ab65c48db820124d75ad0956e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b408a99195b4ef7a317186ba03592c6" is_pre_election: true
I20260929 02:59:54.145274 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e47f2822a70418092991cae5e011c12" candidate_uuid: "c192c27ab65c48db820124d75ad0956e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c7670915d34d478c0079976d48bb66" is_pre_election: true
I20260929 02:59:54.145481 924 raft_consensus.cc:2454] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c192c27ab65c48db820124d75ad0956e in term 0.
I20260929 02:59:54.145787 1056 raft_consensus.cc:2454] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c192c27ab65c48db820124d75ad0956e in term 0.
I20260929 02:59:54.146093 725 leader_election.cc:302] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b408a99195b4ef7a317186ba03592c6, c192c27ab65c48db820124d75ad0956e; no voters:
I20260929 02:59:54.146452 1128 raft_consensus.cc:2781] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:59:54.146600 1128 raft_consensus.cc:482] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:59:54.146721 1128 raft_consensus.cc:3037] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:54.148872 1128 raft_consensus.cc:504] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 } } peers { permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 } } peers { permanent_uuid: "c192c27ab65c48db820124d75ad0956e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44979 } }
I20260929 02:59:54.149502 1128 leader_election.cc:288] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [CANDIDATE]: Term 1 election: Requested vote from peers 6b408a99195b4ef7a317186ba03592c6 (127.15.18.66:43993), 61c7670915d34d478c0079976d48bb66 (127.15.18.67:44969)
I20260929 02:59:54.149901 924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e47f2822a70418092991cae5e011c12" candidate_uuid: "c192c27ab65c48db820124d75ad0956e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b408a99195b4ef7a317186ba03592c6"
I20260929 02:59:54.149986 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e47f2822a70418092991cae5e011c12" candidate_uuid: "c192c27ab65c48db820124d75ad0956e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c7670915d34d478c0079976d48bb66"
I20260929 02:59:54.150085 924 raft_consensus.cc:3037] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:54.150174 1056 raft_consensus.cc:3037] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:54.152379 924 raft_consensus.cc:2454] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c192c27ab65c48db820124d75ad0956e in term 1.
I20260929 02:59:54.152396 1056 raft_consensus.cc:2454] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c192c27ab65c48db820124d75ad0956e in term 1.
I20260929 02:59:54.152756 725 leader_election.cc:302] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61c7670915d34d478c0079976d48bb66, c192c27ab65c48db820124d75ad0956e; no voters:
I20260929 02:59:54.153069 1128 raft_consensus.cc:2781] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:59:54.153502 1128 raft_consensus.cc:686] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 1 LEADER]: Becoming Leader. State: Replica: c192c27ab65c48db820124d75ad0956e, State: Running, Role: LEADER
I20260929 02:59:54.154042 1128 consensus_queue.cc:237] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 } } peers { permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 } } peers { permanent_uuid: "c192c27ab65c48db820124d75ad0956e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44979 } }
I20260929 02:59:54.158823 651 catalog_manager.cc:5495] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e reported cstate change: term changed from 0 to 1, leader changed from <none> to c192c27ab65c48db820124d75ad0956e (127.15.18.65). New cstate: current_term: 1 leader_uuid: "c192c27ab65c48db820124d75ad0956e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c192c27ab65c48db820124d75ad0956e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 44979 } health_report { overall_health: HEALTHY } } }
I20260929 02:59:54.174189 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.168987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.15.18.65:44979)
0929 02:59:54.169191 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:54.169195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:54.169216 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:59:54.169322 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:54.169325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:54.169435 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:54.169556 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:59:54.169840 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.170551 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.171976 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.172411 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.172531 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.173054 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:59:54.173567 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:59:54.173714 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:54.173826 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 02:59:54.173934 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:59:54.183261 924 raft_consensus.cc:1261] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6 [term 1 FOLLOWER]: Refusing update from remote peer c192c27ab65c48db820124d75ad0956e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:59:54.183260 1056 raft_consensus.cc:1261] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66 [term 1 FOLLOWER]: Refusing update from remote peer c192c27ab65c48db820124d75ad0956e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:59:54.184322 1131 consensus_queue.cc:1035] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b408a99195b4ef7a317186ba03592c6" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 43993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:59:54.184738 1128 consensus_queue.cc:1035] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c7670915d34d478c0079976d48bb66" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 44969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:59:54.186444 1102 tablet.cc:2369] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:59:54.216427 650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:59:54.216809 650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:59:54.222604 650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a16cd981e054ffeba202d620a7a3a20: Sending DeleteTablet for 3 replicas of tablet 3e47f2822a70418092991cae5e011c12
I20260929 02:59:54.224144 772 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e47f2822a70418092991cae5e011c12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:54 UTC) from {username='slave'} at 127.0.0.1:34798
I20260929 02:59:54.224213 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e47f2822a70418092991cae5e011c12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:54 UTC) from {username='slave'} at 127.0.0.1:46594
I20260929 02:59:54.224622 904 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e47f2822a70418092991cae5e011c12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:54 UTC) from {username='slave'} at 127.0.0.1:42216
I20260929 02:59:54.224890 903 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e47f2822a70418092991cae5e011c12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:54 UTC) from {username='slave'} at 127.0.0.1:42216
I20260929 02:59:54.226408 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 710
I20260929 02:59:54.235761 1146 tablet_replica.cc:330] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: stopping tablet replica
I20260929 02:59:54.236394 1146 raft_consensus.cc:2229] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:59:54.237097 1146 raft_consensus.cc:2258] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:54.238137 1145 tablet_replica.cc:330] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: stopping tablet replica
I20260929 02:59:54.238672 1145 raft_consensus.cc:2229] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:59:54.239135 1145 raft_consensus.cc:2258] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:54.239602 1144 tablet_replica.cc:330] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e: stopping tablet replica
I20260929 02:59:54.240080 1144 raft_consensus.cc:2229] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:59:54.240415 1146 ts_tablet_manager.cc:1905] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:59:54.240804 1144 raft_consensus.cc:2258] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:54.243052 1144 ts_tablet_manager.cc:1905] T 3e47f2822a70418092991cae5e011c12 P c192c27ab65c48db820124d75ad0956e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:59:54.246860 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.228580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.15.18.65:44979)
0929 02:59:54.228762 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:54.228770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:54.238378 (+ 9608us) server_negotiation.cc:407] Connection header received
0929 02:59:54.238596 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:54.238602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:54.238718 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:54.238833 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:54.239343 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.240577 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.242563 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.243251 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.243360 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.244894 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:54.244904 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:54.244924 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:59:54.245825 (+ 901us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:54.245901 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:59:54.245925 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:54.246013 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:54.246267 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 02:59:54.246392 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:59:54.247767 1146 ts_tablet_manager.cc:1918] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:59:54.247946 1146 log.cc:1196] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/3e47f2822a70418092991cae5e011c12
I20260929 02:59:54.248044 833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:54.248524 1146 ts_tablet_manager.cc:1939] T 3e47f2822a70418092991cae5e011c12 P 6b408a99195b4ef7a317186ba03592c6: Deleting consensus metadata
I20260929 02:59:54.250757 1145 ts_tablet_manager.cc:1905] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:59:54.250844 634 catalog_manager.cc:4841] TS 6b408a99195b4ef7a317186ba03592c6 (127.15.18.66:43993): tablet 3e47f2822a70418092991cae5e011c12 (table test-table [id=7e5072e631ca45a4b3f01f9e64c13f51]) successfully deleted
W20260929 02:59:54.252075 634 catalog_manager.cc:4808] TS 6b408a99195b4ef7a317186ba03592c6 (127.15.18.66:43993): delete failed for tablet 3e47f2822a70418092991cae5e011c12 because the tablet was not found. No further retry: Not found: Tablet not found: 3e47f2822a70418092991cae5e011c12
I20260929 02:59:54.258015 1145 ts_tablet_manager.cc:1918] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:59:54.258193 1145 log.cc:1196] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/3e47f2822a70418092991cae5e011c12
I20260929 02:59:54.258687 1145 ts_tablet_manager.cc:1939] T 3e47f2822a70418092991cae5e011c12 P 61c7670915d34d478c0079976d48bb66: Deleting consensus metadata
I20260929 02:59:54.260675 634 catalog_manager.cc:4841] TS 61c7670915d34d478c0079976d48bb66 (127.15.18.67:44969): tablet 3e47f2822a70418092991cae5e011c12 (table test-table [id=7e5072e631ca45a4b3f01f9e64c13f51]) successfully deleted
I20260929 02:59:54.350116 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 710
I20260929 02:59:54.365274 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 842
W20260929 02:59:54.365386 634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.18.65:44979 (error 108)
I20260929 02:59:54.367166 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.366739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44979 (local address 127.0.0.1:34828)
0929 02:59:54.366927 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:54.367026 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 02:59:54.367422 634 catalog_manager.cc:4584] TS c192c27ab65c48db820124d75ad0956e (127.15.18.65:44979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e47f2822a70418092991cae5e011c12: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44979: connect: Connection refused (error 111)
I20260929 02:59:54.372310 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.366129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.15.18.66:43993)
0929 02:59:54.366326 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:54.366332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:54.366444 (+ 112us) server_negotiation.cc:407] Connection header received
0929 02:59:54.366631 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:54.366635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:54.366742 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:54.366832 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:54.367269 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.368132 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.369440 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.369845 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.369955 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.370933 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:54.370942 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:54.370957 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:59:54.371583 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:54.371636 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:59:54.371657 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:54.371763 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:54.371887 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 02:59:54.371999 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:59:54.372936 965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:54.421022 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.420522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44979 (local address 127.0.0.1:34834)
0929 02:59:54.420743 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:54.420844 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:59:54.422176 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.421775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44979 (local address 127.0.0.1:34842)
0929 02:59:54.421962 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:54.422044 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 02:59:54.461361 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 842
I20260929 02:59:54.474910 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 973
I20260929 02:59:54.482072 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.475886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.15.18.67:44969)
0929 02:59:54.476083 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:54.476088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:54.476106 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:59:54.476158 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:54.476161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:54.476248 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:54.476343 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:54.476742 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.477471 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.478722 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:54.479108 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:54.479214 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:54.480554 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:54.480564 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:54.480588 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:59:54.481269 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:54.481328 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:59:54.481352 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:54.481477 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:54.481610 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 02:59:54.481776 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:59:54.482725 1096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:54.482872 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.482428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44979 (local address 127.0.0.1:34856)
0929 02:59:54.482613 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:54.482700 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 02:59:54.484220 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.483892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44979 (local address 127.0.0.1:34866)
0929 02:59:54.484017 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:54.484085 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 02:59:54.565217 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.564617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44979 (local address 127.0.0.1:34874)
0929 02:59:54.564899 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:54.565030 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 02:59:54.566502 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:54.566037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:44979 (local address 127.0.0.1:34884)
0929 02:59:54.566280 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:54.566369 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 02:59:54.573263 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 973
I20260929 02:59:54.586290 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 619
I20260929 02:59:54.587386 681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:54.696794 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 619
2026-09-29T02:59:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2711 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:59:54 dist-test-slave-q5f0 krb5kdc[1155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:59:54 dist-test-slave-q5f0 krb5kdc[1155](info): set up 2 sockets
Sep 29 02:59:54 dist-test-slave-q5f0 krb5kdc[1155](info): commencing operation
krb5kdc: starting...
W20260929 02:59:55.719890 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 02:59:55 dist-test-slave-q5f0 krb5kdc[1155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:59:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:59:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 02:59:55.957118 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:42215
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:42215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:59:56.279448 1168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:56.279772 1168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:56.279912 1168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:56.287217 1168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:59:56.287338 1168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:56.287388 1168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:56.287469 1168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:59:56.287554 1168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:59:56.297561 1168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:42215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:42215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:56.298874 1168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:56.300217 1168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:56.311585 1173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:56.311659 1174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:56.311676 1176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:56.312436 1168 server_base.cc:1034] running on GCE node
I20260929 02:59:56.313164 1168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:56.314553 1168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:56.315743 1168 hybrid_clock.cc:648] HybridClock initialized: now 1790650796315719 us; error 52 us; skew 500 ppm
I20260929 02:59:56.316107 1168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:56 dist-test-slave-q5f0 krb5kdc[1155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:56.319170 1168 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 02:59:56.320776 1168 webserver.cc:460] Webserver started at http://127.15.18.126:44027/ using document root <none> and password file <none>
I20260929 02:59:56.321458 1168 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:56.321578 1168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:56.321872 1168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:56.324946 1168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "6c869ef6a49a4f549971fc05430ba365"
format_stamp: "Formatted at 2026-09-29 02:59:56 on dist-test-slave-q5f0"
server_key: "b5a0e6d5b5f88440caa746c3f2f51496"
server_key_iv: "0a3e811c58c7eb4a2143946daf3aa553"
server_key_version: "encryptionkey@0"
I20260929 02:59:56.325672 1168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "6c869ef6a49a4f549971fc05430ba365"
format_stamp: "Formatted at 2026-09-29 02:59:56 on dist-test-slave-q5f0"
server_key: "b5a0e6d5b5f88440caa746c3f2f51496"
server_key_iv: "0a3e811c58c7eb4a2143946daf3aa553"
server_key_version: "encryptionkey@0"
I20260929 02:59:56.331650 1168 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 02:59:56.335111 1183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:56.336635 1168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:59:56.336853 1168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "6c869ef6a49a4f549971fc05430ba365"
format_stamp: "Formatted at 2026-09-29 02:59:56 on dist-test-slave-q5f0"
server_key: "b5a0e6d5b5f88440caa746c3f2f51496"
server_key_iv: "0a3e811c58c7eb4a2143946daf3aa553"
server_key_version: "encryptionkey@0"
I20260929 02:59:56.337042 1168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:56.362977 1168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:56.367789 1168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:56.368074 1168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:56.385809 1168 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:42215
I20260929 02:59:56.385842 1234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:42215 every 8 connection(s)
I20260929 02:59:56.387236 1168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:59:56.389918 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 1168
I20260929 02:59:56.390105 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:59:56.390535 15433 external_mini_cluster.cc:1428] Setting key 9f8accff9fd2ae6ae08d6ce9d8df3ebc
I20260929 02:59:56.391426 1235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:59:56 dist-test-slave-q5f0 krb5kdc[1155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:56.406018 1235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365: Bootstrap starting.
I20260929 02:59:56.409515 1235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:56.410792 1235 log.cc:824] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:56.414131 1235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365: No bootstrap required, opened a new log
I20260929 02:59:56.414463 1238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:56.392514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.15.18.126:42215)
0929 02:59:56.393314 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:56.393326 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:56.393369 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:59:56.394912 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:56.394934 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:56.395339 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:56.395729 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:56.397557 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:56.398962 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:56.400837 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:56.401539 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:56.401689 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:56.406564 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:56.406573 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:56.406589 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:56.406631 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:56.409861 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:56.410765 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:56.410772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:56.410783 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:56.410981 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:56.411460 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:56.411464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:56.411467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:56.412224 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:56.412474 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:56.412991 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 02:59:56.413399 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":352,"threads_started":1}
I20260929 02:59:56.421553 1235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c869ef6a49a4f549971fc05430ba365" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 42215 } }
I20260929 02:59:56.421938 1235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:56.422040 1235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c869ef6a49a4f549971fc05430ba365, State: Initialized, Role: FOLLOWER
I20260929 02:59:56.422698 1235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c869ef6a49a4f549971fc05430ba365" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 42215 } }
I20260929 02:59:56.422947 1235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:59:56.423077 1235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:59:56.423241 1235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:56.425447 1235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c869ef6a49a4f549971fc05430ba365" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 42215 } }
I20260929 02:59:56.425959 1235 leader_election.cc:302] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c869ef6a49a4f549971fc05430ba365; no voters:
I20260929 02:59:56.426534 1235 leader_election.cc:288] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:59:56.426679 1240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:59:56.427407 1240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [term 1 LEADER]: Becoming Leader. State: Replica: 6c869ef6a49a4f549971fc05430ba365, State: Running, Role: LEADER
I20260929 02:59:56.428027 1240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c869ef6a49a4f549971fc05430ba365" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 42215 } }
I20260929 02:59:56.428663 1235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:59:56.430981 1242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c869ef6a49a4f549971fc05430ba365. Latest consensus state: current_term: 1 leader_uuid: "6c869ef6a49a4f549971fc05430ba365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c869ef6a49a4f549971fc05430ba365" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 42215 } } }
I20260929 02:59:56.431546 1242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:56.431169 1241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c869ef6a49a4f549971fc05430ba365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c869ef6a49a4f549971fc05430ba365" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 42215 } } }
I20260929 02:59:56.432039 1241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365 [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:56.433588 1249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:59:56.437814 1249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:59:56.446498 1249 catalog_manager.cc:1347] Generated new cluster ID: cc4701de26d94e95b7ba710eca85f99c
I20260929 02:59:56.446619 1249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:59:56.465318 1249 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:59:56.466665 1249 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:59:56.474593 1249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365: Generated new TSK 0
I20260929 02:59:56.475472 1249 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 02:59:56.534632 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:42215
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:59:56.873442 1261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:56.873750 1261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:56.873852 1261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:56.881268 1261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:56.881408 1261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:56.881543 1261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 02:59:56.891644 1261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:42215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:56.892957 1261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:56.894279 1261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:56.907721 1270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:56.907835 1267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:56.907748 1268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:56.908418 1261 server_base.cc:1034] running on GCE node
I20260929 02:59:56.909029 1261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:56.909991 1261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:56.911209 1261 hybrid_clock.cc:648] HybridClock initialized: now 1790650796911153 us; error 42 us; skew 500 ppm
I20260929 02:59:56.911558 1261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:56 dist-test-slave-q5f0 krb5kdc[1155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:56.919441 1261 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 02:59:56.921101 1261 webserver.cc:460] Webserver started at http://127.15.18.65:41647/ using document root <none> and password file <none>
I20260929 02:59:56.921844 1261 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:56.921969 1261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:56.922279 1261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:56.925508 1261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224"
format_stamp: "Formatted at 2026-09-29 02:59:56 on dist-test-slave-q5f0"
server_key: "df86f38ea2f1a175dc206bab8f3b929f"
server_key_iv: "4e134d2d82e78a71f50c9f2599b7ffe2"
server_key_version: "encryptionkey@0"
I20260929 02:59:56.926343 1261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224"
format_stamp: "Formatted at 2026-09-29 02:59:56 on dist-test-slave-q5f0"
server_key: "df86f38ea2f1a175dc206bab8f3b929f"
server_key_iv: "4e134d2d82e78a71f50c9f2599b7ffe2"
server_key_version: "encryptionkey@0"
I20260929 02:59:56.932081 1261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:59:56.936015 1277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:56.937541 1261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:59:56.937731 1261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224"
format_stamp: "Formatted at 2026-09-29 02:59:56 on dist-test-slave-q5f0"
server_key: "df86f38ea2f1a175dc206bab8f3b929f"
server_key_iv: "4e134d2d82e78a71f50c9f2599b7ffe2"
server_key_version: "encryptionkey@0"
I20260929 02:59:56.937952 1261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:56.973937 1261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:56.978729 1261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:56.979020 1261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:56.980057 1261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:56.981472 1261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:56.981554 1261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:56.981667 1261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:56.981732 1261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:57.012480 1261 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:43719
I20260929 02:59:57.012532 1389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:43719 every 8 connection(s)
I20260929 02:59:57.013733 1261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 02:59:57.018855 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 1261
I20260929 02:59:57.019050 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 02:59:57.019534 15433 external_mini_cluster.cc:1428] Setting key f5acd9a488db8b5ff60a4181a511b8b5
Sep 29 02:59:57 dist-test-slave-q5f0 krb5kdc[1155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650796, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:57.036521 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:57.016231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:46129 (local address 127.15.18.126:42215)
0929 02:59:57.016682 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:57.016688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:57.017747 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 02:59:57.019675 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:57.019683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:57.019789 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:57.019960 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:57.021835 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:57.022883 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:57.024751 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:57.025345 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:57.025482 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:57.029429 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:57.029435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:57.029438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:57.029503 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:57.033057 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:57.034049 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:57.034054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:57.034057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:57.034197 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:57.034647 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:57.034650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:57.034652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:57.034947 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:57.035084 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:57.035906 (+ 822us) server_negotiation.cc:299] Negotiation successful
0929 02:59:57.036144 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":206,"threads_started":1}
I20260929 02:59:57.037498 1393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:57.016902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:42215 (local address 127.15.18.65:46129)
0929 02:59:57.017576 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:57.017621 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:57.019358 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:57.020238 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:57.020248 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:57.020942 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:57.021645 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:57.021668 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:57.023087 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:57.023093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:57.024595 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:57.024608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:57.025608 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:57.025614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:57.025833 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:57.026569 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:57.026598 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:57.029126 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:57.033283 (+ 4157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:57.033295 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:57.033314 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:57.033871 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:57.034311 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:57.034315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:57.034317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:57.034521 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:57.035077 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:57.035085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:57.035473 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 02:59:57.035777 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 02:59:57.036356 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
I20260929 02:59:57.038995 1390 heartbeater.cc:347] Connected to a master server at 127.15.18.126:42215
I20260929 02:59:57.039364 1390 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:57.040266 1390 heartbeater.cc:511] Master 127.15.18.126:42215 requested a full tablet report, sending...
I20260929 02:59:57.042624 1200 ts_manager.cc:194] Registered new tserver with Master: 2cdd8d914ad84edcbe33ac2fbe9a7224 (127.15.18.65:43719)
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
I20260929 02:59:57.044162 1200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:46129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 02:59:57.074615 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:42215
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:59:57.396698 1396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:57.397060 1396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:57.397192 1396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:57.408505 1396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:57.408643 1396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:57.408771 1396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 02:59:57.418617 1396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:42215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:57.420073 1396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:57.421440 1396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:57.436291 1402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:57.436225 1404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:57.436508 1401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:57.436942 1396 server_base.cc:1034] running on GCE node
I20260929 02:59:57.437601 1396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:57.438577 1396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:57.439816 1396 hybrid_clock.cc:648] HybridClock initialized: now 1790650797439767 us; error 37 us; skew 500 ppm
I20260929 02:59:57.440145 1396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:57 dist-test-slave-q5f0 krb5kdc[1155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:57.443315 1396 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 02:59:57.444765 1396 webserver.cc:460] Webserver started at http://127.15.18.66:38181/ using document root <none> and password file <none>
I20260929 02:59:57.445411 1396 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:57.445506 1396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:57.445768 1396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:57.448643 1396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "bb418f3721aa4b3c94e3671734ddf124"
format_stamp: "Formatted at 2026-09-29 02:59:57 on dist-test-slave-q5f0"
server_key: "06e062d806d6027c17a4c67842f4e5a8"
server_key_iv: "d53ac8798bc4eeee62627d3565b9ba29"
server_key_version: "encryptionkey@0"
I20260929 02:59:57.449340 1396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "bb418f3721aa4b3c94e3671734ddf124"
format_stamp: "Formatted at 2026-09-29 02:59:57 on dist-test-slave-q5f0"
server_key: "06e062d806d6027c17a4c67842f4e5a8"
server_key_iv: "d53ac8798bc4eeee62627d3565b9ba29"
server_key_version: "encryptionkey@0"
I20260929 02:59:57.454402 1396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:59:57.457789 1411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:57.459158 1396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:59:57.459321 1396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "bb418f3721aa4b3c94e3671734ddf124"
format_stamp: "Formatted at 2026-09-29 02:59:57 on dist-test-slave-q5f0"
server_key: "06e062d806d6027c17a4c67842f4e5a8"
server_key_iv: "d53ac8798bc4eeee62627d3565b9ba29"
server_key_version: "encryptionkey@0"
I20260929 02:59:57.459584 1396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:57.490773 1396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:57.495100 1396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:57.495378 1396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:57.496457 1396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:57.497815 1396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:57.497897 1396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:57.498013 1396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:57.498070 1396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:57.527566 1396 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:36585
I20260929 02:59:57.527650 1524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:36585 every 8 connection(s)
I20260929 02:59:57.528992 1396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 02:59:57.538096 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 1396
I20260929 02:59:57.538295 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 02:59:57.538728 15433 external_mini_cluster.cc:1428] Setting key 2cca48f22cfc28563d8eec5268decf82
Sep 29 02:59:57 dist-test-slave-q5f0 krb5kdc[1155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:57.548810 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:57.531616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:39115 (local address 127.15.18.126:42215)
0929 02:59:57.531800 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:57.531804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:57.533128 (+ 1324us) server_negotiation.cc:407] Connection header received
0929 02:59:57.534963 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:57.534968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:57.535044 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:57.535134 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:57.536680 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:57.537386 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:57.539122 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:57.539680 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:57.539802 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:57.543314 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:57.543319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:57.543323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:57.543357 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:57.545826 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:57.546552 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:57.546557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:57.546559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:57.546688 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:57.547102 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:57.547106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:57.547108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:57.547424 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:57.547607 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:57.548327 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 02:59:57.548554 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:59:57.549777 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:57.532148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:42215 (local address 127.15.18.66:39115)
0929 02:59:57.532941 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:57.532994 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:57.534690 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:57.535342 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:57.535351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:57.535962 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:57.536526 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:57.536543 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:57.537533 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:57.537537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:57.538979 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:57.538990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:57.539895 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:57.539900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:57.540097 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:57.540781 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:57.540812 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:57.543093 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:57.545972 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:57.545983 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:57.546001 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:57.546437 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:57.546792 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:57.546797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:57.546801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:57.546996 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:57.547599 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:57.547606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:57.547970 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 02:59:57.548217 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 02:59:57.548641 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":357,"threads_started":1}
I20260929 02:59:57.551267 1525 heartbeater.cc:347] Connected to a master server at 127.15.18.126:42215
I20260929 02:59:57.551637 1525 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:57.552451 1525 heartbeater.cc:511] Master 127.15.18.126:42215 requested a full tablet report, sending...
I20260929 02:59:57.553972 1200 ts_manager.cc:194] Registered new tserver with Master: bb418f3721aa4b3c94e3671734ddf124 (127.15.18.66:36585)
I20260929 02:59:57.554757 1200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:39115
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 02:59:57.591312 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:42215
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:59:57.910166 1530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:57.910455 1530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:57.910578 1530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:57.917896 1530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:57.918016 1530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:57.918159 1530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 02:59:57.928113 1530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:42215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:57.929478 1530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:57.930799 1530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:57.944093 1538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:57.944031 1536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:57.944685 1535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:57.944748 1530 server_base.cc:1034] running on GCE node
I20260929 02:59:57.945397 1530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:57.946341 1530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:57.947597 1530 hybrid_clock.cc:648] HybridClock initialized: now 1790650797947538 us; error 42 us; skew 500 ppm
I20260929 02:59:57.947909 1530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:59:57 dist-test-slave-q5f0 krb5kdc[1155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:59:57.950973 1530 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 02:59:57.952489 1530 webserver.cc:460] Webserver started at http://127.15.18.67:36271/ using document root <none> and password file <none>
I20260929 02:59:57.953151 1530 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:57.953248 1530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:57.953503 1530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:57.956475 1530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "c9a9a47ad9d04d02ae767b32e7ed7272"
format_stamp: "Formatted at 2026-09-29 02:59:57 on dist-test-slave-q5f0"
server_key: "7a8d37f5e0f061c359621b177233f793"
server_key_iv: "c21ac5cf39c7ea7ea2bdc3d9a57bd9aa"
server_key_version: "encryptionkey@0"
I20260929 02:59:57.957245 1530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "c9a9a47ad9d04d02ae767b32e7ed7272"
format_stamp: "Formatted at 2026-09-29 02:59:57 on dist-test-slave-q5f0"
server_key: "7a8d37f5e0f061c359621b177233f793"
server_key_iv: "c21ac5cf39c7ea7ea2bdc3d9a57bd9aa"
server_key_version: "encryptionkey@0"
I20260929 02:59:57.962473 1530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:59:57.966156 1545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:57.967607 1530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:59:57.967763 1530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "c9a9a47ad9d04d02ae767b32e7ed7272"
format_stamp: "Formatted at 2026-09-29 02:59:57 on dist-test-slave-q5f0"
server_key: "7a8d37f5e0f061c359621b177233f793"
server_key_iv: "c21ac5cf39c7ea7ea2bdc3d9a57bd9aa"
server_key_version: "encryptionkey@0"
I20260929 02:59:57.967955 1530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:57.997638 1530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:58.002105 1530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:58.002401 1530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:58.003427 1530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:59:58.004827 1530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:59:58.004904 1530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:58.005025 1530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:59:58.005084 1530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:58.036628 1530 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:33253
I20260929 02:59:58.036679 1658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:33253 every 8 connection(s)
I20260929 02:59:58.037882 1530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 02:59:58.044824 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 1530
I20260929 02:59:58.045017 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 02:59:58.045612 15433 external_mini_cluster.cc:1428] Setting key 50a71ddfcada4be97348313d5819ddb9
I20260929 02:59:58.047356 1390 heartbeater.cc:503] Master 127.15.18.126:42215 was elected leader, sending a full tablet report...
Sep 29 02:59:58 dist-test-slave-q5f0 krb5kdc[1155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 02:59:58.057829 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.040420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:56231 (local address 127.15.18.126:42215)
0929 02:59:58.040626 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.040630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.041770 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 02:59:58.043720 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.043724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.043821 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.043905 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:58.045926 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.046838 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.048992 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.049398 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.049502 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.052745 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:58.052750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:58.052753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:58.052784 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:58.055175 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:58.055922 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:58.055926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:58.055928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:58.056052 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:58.056416 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:58.056419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:58.056421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:58.056672 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:58.056778 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.057410 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.057572 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:59:58.058848 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.040903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:42215 (local address 127.15.18.67:56231)
0929 02:59:58.041595 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:58.041652 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:58.043417 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:58.044174 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:58.044183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:58.044904 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:59:58.045727 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.045755 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.046985 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.046989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.048860 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.048872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.049610 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.049616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.049848 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.050421 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:59:58.050443 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:59:58.052560 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:59:58.055324 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:58.055331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:58.055344 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:58.055824 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:58.056149 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:59:58.056152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:59:58.056154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:59:58.056338 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:59:58.056798 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:59:58.056804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:59:58.057112 (+ 308us) client_negotiation.cc:769] Sending connection context
0929 02:59:58.057335 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 02:59:58.057721 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":290,"threads_started":1}
I20260929 02:59:58.060340 1659 heartbeater.cc:347] Connected to a master server at 127.15.18.126:42215
I20260929 02:59:58.060642 1659 heartbeater.cc:464] Registering TS with master...
I20260929 02:59:58.061372 1659 heartbeater.cc:511] Master 127.15.18.126:42215 requested a full tablet report, sending...
I20260929 02:59:58.062654 1199 ts_manager.cc:194] Registered new tserver with Master: c9a9a47ad9d04d02ae767b32e7ed7272 (127.15.18.67:33253)
I20260929 02:59:58.063606 1199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:56231
I20260929 02:59:58.073671 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:59:58.089931 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.077819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.15.18.126:42215)
0929 02:59:58.077996 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.078001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.078408 (+ 407us) server_negotiation.cc:407] Connection header received
0929 02:59:58.078556 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.078559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.078622 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.078716 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:58.080167 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.080902 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.082133 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.082521 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.082610 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.085043 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:58.085048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:58.085052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:58.085085 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:58.087579 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:58.088301 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:58.088304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:58.088307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:58.088445 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:58.088802 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:58.088805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:58.088807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:58.089098 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:58.089293 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.089570 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.089706 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:59:58.093822 1199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:59:58.100703 1199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:59:58.124317 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.113143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:33253 (local address 127.0.0.1:48958)
0929 02:59:58.113924 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:58.113979 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:58.114145 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:58.117243 (+ 3098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:58.117251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:58.117282 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:58.117710 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.117723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.119667 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.119672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.121860 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.121875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.123316 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.123323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.123568 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.123618 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 02:59:58.123724 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 02:59:58.123939 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":370,"threads_started":1}
I20260929 02:59:58.125716 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.113306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.15.18.67:33253)
0929 02:59:58.116569 (+ 3263us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.116578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.116613 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:59:58.116708 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.116716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.117006 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.117927 (+ 921us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:58.118192 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.119530 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.122026 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.123055 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.124868 (+ 1813us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.125102 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.125184 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.125374 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3080,"thread_start_us":3986,"threads_started":1}
I20260929 02:59:58.126606 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.113887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:36585 (local address 127.0.0.1:43926)
0929 02:59:58.114840 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:58.114867 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:58.114974 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:58.118849 (+ 3875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:58.118854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:58.118880 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:58.119306 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.119315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.121350 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.121356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.123592 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.123605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.125799 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.125806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.126021 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.126053 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:59:58.126129 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 02:59:58.126261 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":57,"thread_start_us":1792,"threads_started":1}
I20260929 02:59:58.126709 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.114859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.15.18.65:43719)
0929 02:59:58.117992 (+ 3133us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.118001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.118029 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:59:58.118136 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.118144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.118426 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.118640 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:58.119291 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.120741 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.123439 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.124707 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.124837 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.125072 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.126180 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.126362 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2966,"thread_start_us":2800,"threads_started":1}
I20260929 02:59:58.127038 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.114418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.15.18.66:36585)
0929 02:59:58.118316 (+ 3898us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.118323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.118345 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:59:58.118444 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.118449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.118631 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.119458 (+ 827us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:58.119694 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.121196 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.123754 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.125570 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.125718 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.126198 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.126280 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.126423 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3751,"thread_start_us":3624,"threads_started":1}
I20260929 02:59:58.127059 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.114335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:43719 (local address 127.0.0.1:56080)
0929 02:59:58.115931 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:58.115949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:58.116049 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:58.118634 (+ 2585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:58.118639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:58.118664 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:58.119083 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.119094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.120896 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.120902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.123232 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.123248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.125219 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.125226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.125392 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.125422 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:59:58.125510 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:59:58.125626 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1479,"mutex_wait_us":98,"thread_start_us":694,"threads_started":1}
I20260929 02:59:58.130050 1594 tablet_service.cc:1467] Processing CreateTablet for tablet fafb303405c24a8c85233fe90e0517a3 (DEFAULT_TABLE table=test-table [id=dcff7aff38cb4a459bab7fe152c55fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:58.130498 1325 tablet_service.cc:1467] Processing CreateTablet for tablet fafb303405c24a8c85233fe90e0517a3 (DEFAULT_TABLE table=test-table [id=dcff7aff38cb4a459bab7fe152c55fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:58.130546 1460 tablet_service.cc:1467] Processing CreateTablet for tablet fafb303405c24a8c85233fe90e0517a3 (DEFAULT_TABLE table=test-table [id=dcff7aff38cb4a459bab7fe152c55fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:59:58.131749 1594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafb303405c24a8c85233fe90e0517a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:58.131754 1460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafb303405c24a8c85233fe90e0517a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:58.131853 1325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafb303405c24a8c85233fe90e0517a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:58.146221 1679 tablet_bootstrap.cc:492] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: Bootstrap starting.
I20260929 02:59:58.148566 1677 tablet_bootstrap.cc:492] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: Bootstrap starting.
I20260929 02:59:58.149566 1678 tablet_bootstrap.cc:492] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: Bootstrap starting.
I20260929 02:59:58.151044 1679 tablet_bootstrap.cc:654] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:58.151902 1677 tablet_bootstrap.cc:654] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:58.152424 1679 log.cc:824] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:58.152998 1678 tablet_bootstrap.cc:654] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:58.152997 1677 log.cc:824] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:58.154368 1678 log.cc:824] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:58.157289 1678 tablet_bootstrap.cc:492] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: No bootstrap required, opened a new log
I20260929 02:59:58.157543 1678 ts_tablet_manager.cc:1397] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:59:58.157560 1677 tablet_bootstrap.cc:492] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: No bootstrap required, opened a new log
I20260929 02:59:58.157829 1677 ts_tablet_manager.cc:1397] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 02:59:58.157895 1679 tablet_bootstrap.cc:492] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: No bootstrap required, opened a new log
I20260929 02:59:58.158187 1679 ts_tablet_manager.cc:1397] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260929 02:59:58.165256 1678 raft_consensus.cc:348] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33253 } } peers { permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 } } peers { permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 } }
I20260929 02:59:58.165642 1678 raft_consensus.cc:374] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:58.165757 1678 raft_consensus.cc:729] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb418f3721aa4b3c94e3671734ddf124, State: Initialized, Role: FOLLOWER
I20260929 02:59:58.166349 1678 consensus_queue.cc:260] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33253 } } peers { permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 } } peers { permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 } }
I20260929 02:59:58.167421 1525 heartbeater.cc:503] Master 127.15.18.126:42215 was elected leader, sending a full tablet report...
I20260929 02:59:58.168167 1678 ts_tablet_manager.cc:1428] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 02:59:58.168422 1679 raft_consensus.cc:348] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33253 } } peers { permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 } } peers { permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 } }
I20260929 02:59:58.168422 1677 raft_consensus.cc:348] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33253 } } peers { permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 } } peers { permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 } }
I20260929 02:59:58.168871 1677 raft_consensus.cc:374] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:58.168871 1679 raft_consensus.cc:374] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:58.168975 1679 raft_consensus.cc:729] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a9a47ad9d04d02ae767b32e7ed7272, State: Initialized, Role: FOLLOWER
I20260929 02:59:58.168972 1677 raft_consensus.cc:729] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdd8d914ad84edcbe33ac2fbe9a7224, State: Initialized, Role: FOLLOWER
I20260929 02:59:58.169611 1679 consensus_queue.cc:260] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33253 } } peers { permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 } } peers { permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 } }
I20260929 02:59:58.169611 1677 consensus_queue.cc:260] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33253 } } peers { permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 } } peers { permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 } }
I20260929 02:59:58.170749 1659 heartbeater.cc:503] Master 127.15.18.126:42215 was elected leader, sending a full tablet report...
I20260929 02:59:58.171378 1679 ts_tablet_manager.cc:1428] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260929 02:59:58.172868 1677 ts_tablet_manager.cc:1428] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260929 02:59:58.266000 1391 tablet.cc:2369] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:59:58.281138 1526 tablet.cc:2369] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:59:58.289517 1660 tablet.cc:2369] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:59:58.384265 1685 raft_consensus.cc:482] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:59:58.384578 1685 raft_consensus.cc:504] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33253 } } peers { permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 } } peers { permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 } }
I20260929 02:59:58.386373 1685 leader_election.cc:288] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cdd8d914ad84edcbe33ac2fbe9a7224 (127.15.18.65:43719), bb418f3721aa4b3c94e3671734ddf124 (127.15.18.66:36585)
I20260929 02:59:58.393237 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.386892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:48479 (local address 127.15.18.65:43719)
0929 02:59:58.387098 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.387104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.387439 (+ 335us) server_negotiation.cc:407] Connection header received
0929 02:59:58.387562 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.387567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.387651 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.387748 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:58.388295 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.389242 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.391154 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.392371 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.392523 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.392596 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.392872 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.392997 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:59:58.393291 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.386618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:43719 (local address 127.15.18.67:48479)
0929 02:59:58.387161 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:58.387179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:58.387316 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:58.387930 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:58.387935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:58.387965 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:58.388149 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.388160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.389371 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.389375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.390983 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.390998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.392523 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.392529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.392711 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.392746 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:59:58.392846 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:59:58.392979 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":1}
I20260929 02:59:58.394040 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.387259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:58017 (local address 127.15.18.66:36585)
0929 02:59:58.387446 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.387451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.388043 (+ 592us) server_negotiation.cc:407] Connection header received
0929 02:59:58.388212 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.388217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.388295 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.388390 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:59:58.388972 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.390022 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.391955 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.392890 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.393026 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.393095 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.393669 (+ 574us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.393802 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:59:58.394060 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.386862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:36585 (local address 127.15.18.67:58017)
0929 02:59:58.387931 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0929 02:59:58.387946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:59:58.388085 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:59:58.388417 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:59:58.388422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:59:58.388447 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:59:58.388814 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.388826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.390178 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.390184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.391808 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:59:58.391818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.393224 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.393230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:59:58.393527 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.393552 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:59:58.393621 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 02:59:58.393748 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":721,"threads_started":1}
I20260929 02:59:58.394428 1345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb303405c24a8c85233fe90e0517a3" candidate_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" is_pre_election: true
I20260929 02:59:58.395031 1345 raft_consensus.cc:2454] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a9a47ad9d04d02ae767b32e7ed7272 in term 0.
I20260929 02:59:58.395009 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb303405c24a8c85233fe90e0517a3" candidate_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb418f3721aa4b3c94e3671734ddf124" is_pre_election: true
I20260929 02:59:58.395535 1480 raft_consensus.cc:2454] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a9a47ad9d04d02ae767b32e7ed7272 in term 0.
I20260929 02:59:58.395800 1546 leader_election.cc:302] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cdd8d914ad84edcbe33ac2fbe9a7224, c9a9a47ad9d04d02ae767b32e7ed7272; no voters:
I20260929 02:59:58.396195 1685 raft_consensus.cc:2781] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:59:58.396347 1685 raft_consensus.cc:482] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:59:58.396449 1685 raft_consensus.cc:3037] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:58.398563 1685 raft_consensus.cc:504] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33253 } } peers { permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 } } peers { permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 } }
I20260929 02:59:58.399187 1685 leader_election.cc:288] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [CANDIDATE]: Term 1 election: Requested vote from peers 2cdd8d914ad84edcbe33ac2fbe9a7224 (127.15.18.65:43719), bb418f3721aa4b3c94e3671734ddf124 (127.15.18.66:36585)
I20260929 02:59:58.399685 1345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb303405c24a8c85233fe90e0517a3" candidate_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224"
I20260929 02:59:58.399717 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb303405c24a8c85233fe90e0517a3" candidate_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb418f3721aa4b3c94e3671734ddf124"
I20260929 02:59:58.399921 1480 raft_consensus.cc:3037] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:58.399921 1345 raft_consensus.cc:3037] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:58.402980 1345 raft_consensus.cc:2454] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a9a47ad9d04d02ae767b32e7ed7272 in term 1.
I20260929 02:59:58.402980 1480 raft_consensus.cc:2454] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a9a47ad9d04d02ae767b32e7ed7272 in term 1.
I20260929 02:59:58.403409 1546 leader_election.cc:302] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cdd8d914ad84edcbe33ac2fbe9a7224, c9a9a47ad9d04d02ae767b32e7ed7272; no voters:
I20260929 02:59:58.403726 1685 raft_consensus.cc:2781] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:59:58.404166 1685 raft_consensus.cc:686] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 1 LEADER]: Becoming Leader. State: Replica: c9a9a47ad9d04d02ae767b32e7ed7272, State: Running, Role: LEADER
I20260929 02:59:58.404639 1685 consensus_queue.cc:237] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33253 } } peers { permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 } } peers { permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 } }
I20260929 02:59:58.409062 1200 catalog_manager.cc:5495] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a9a47ad9d04d02ae767b32e7ed7272 (127.15.18.67). New cstate: current_term: 1 leader_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9a9a47ad9d04d02ae767b32e7ed7272" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 } health_report { overall_health: UNKNOWN } } }
I20260929 02:59:58.493351 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.487877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.15.18.67:33253)
0929 02:59:58.488097 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.488103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.488180 (+ 77us) server_negotiation.cc:407] Connection header received
0929 02:59:58.488421 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.488425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.488506 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.488633 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:59:58.489046 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.489771 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.491143 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.491572 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.491681 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.492248 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:59:58.492755 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:59:58.492918 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.493069 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.493117 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 02:59:58.501312 1480 raft_consensus.cc:1261] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124 [term 1 FOLLOWER]: Refusing update from remote peer c9a9a47ad9d04d02ae767b32e7ed7272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:59:58.501312 1345 raft_consensus.cc:1261] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224 [term 1 FOLLOWER]: Refusing update from remote peer c9a9a47ad9d04d02ae767b32e7ed7272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:59:58.502300 1687 consensus_queue.cc:1035] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cdd8d914ad84edcbe33ac2fbe9a7224" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 43719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:59:58.502667 1685 consensus_queue.cc:1035] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb418f3721aa4b3c94e3671734ddf124" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:59:58.528971 1198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:59:58.529321 1198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:59:58.534056 1198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c869ef6a49a4f549971fc05430ba365: Sending DeleteTablet for 3 replicas of tablet fafb303405c24a8c85233fe90e0517a3
I20260929 02:59:58.535472 1460 tablet_service.cc:1514] Processing DeleteTablet for tablet fafb303405c24a8c85233fe90e0517a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:58 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:43926
I20260929 02:59:58.535466 1325 tablet_service.cc:1514] Processing DeleteTablet for tablet fafb303405c24a8c85233fe90e0517a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:58 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:56080
I20260929 02:59:58.535672 1594 tablet_service.cc:1514] Processing DeleteTablet for tablet fafb303405c24a8c85233fe90e0517a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:59:58 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:48958
I20260929 02:59:58.540726 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 1261
I20260929 02:59:58.554932 1702 tablet_replica.cc:330] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: stopping tablet replica
I20260929 02:59:58.555300 1703 tablet_replica.cc:330] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: stopping tablet replica
I20260929 02:59:58.555475 1702 raft_consensus.cc:2229] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:59:58.555953 1703 raft_consensus.cc:2229] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:59:58.556054 1702 raft_consensus.cc:2258] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:58.556721 1703 raft_consensus.cc:2258] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:58.558310 1701 tablet_replica.cc:330] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: stopping tablet replica
I20260929 02:59:58.558730 1701 raft_consensus.cc:2229] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:59:58.559235 1701 raft_consensus.cc:2258] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:59:58.559226 1703 ts_tablet_manager.cc:1905] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:59:58.561457 1701 ts_tablet_manager.cc:1905] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:59:58.561796 1702 ts_tablet_manager.cc:1905] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:59:58 dist-test-slave-q5f0 krb5kdc[1155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 02:59:58.568871 1701 ts_tablet_manager.cc:1918] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:59:58.569168 1701 log.cc:1196] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/fafb303405c24a8c85233fe90e0517a3
I20260929 02:59:58.569868 1701 ts_tablet_manager.cc:1939] T fafb303405c24a8c85233fe90e0517a3 P 2cdd8d914ad84edcbe33ac2fbe9a7224: Deleting consensus metadata
I20260929 02:59:58.569993 1702 ts_tablet_manager.cc:1918] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:59:58.570246 1702 log.cc:1196] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/fafb303405c24a8c85233fe90e0517a3
I20260929 02:59:58.570833 1702 ts_tablet_manager.cc:1939] T fafb303405c24a8c85233fe90e0517a3 P bb418f3721aa4b3c94e3671734ddf124: Deleting consensus metadata
I20260929 02:59:58.572225 1184 catalog_manager.cc:4841] TS 2cdd8d914ad84edcbe33ac2fbe9a7224 (127.15.18.65:43719): tablet fafb303405c24a8c85233fe90e0517a3 (table test-table [id=dcff7aff38cb4a459bab7fe152c55fb1]) successfully deleted
I20260929 02:59:58.572667 1186 catalog_manager.cc:4841] TS bb418f3721aa4b3c94e3671734ddf124 (127.15.18.66:36585): tablet fafb303405c24a8c85233fe90e0517a3 (table test-table [id=dcff7aff38cb4a459bab7fe152c55fb1]) successfully deleted
I20260929 02:59:58.574906 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.541994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.15.18.65:43719)
0929 02:59:58.542172 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.542178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.553327 (+ 11149us) server_negotiation.cc:407] Connection header received
0929 02:59:58.553540 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.553547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.553654 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.553764 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:58.556443 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.557838 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.559898 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.560512 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.560639 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.566540 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:58.566557 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:58.566576 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:58.566637 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:58.570698 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:58.572282 (+ 1584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:58.572294 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:58.572297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:58.572467 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:58.573082 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:58.573088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:58.573092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:58.573566 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:58.573908 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.574256 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.574501 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:59:58.575696 1385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:58.579242 1703 ts_tablet_manager.cc:1918] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:59:58.579422 1703 log.cc:1196] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/fafb303405c24a8c85233fe90e0517a3
I20260929 02:59:58.579934 1703 ts_tablet_manager.cc:1939] T fafb303405c24a8c85233fe90e0517a3 P c9a9a47ad9d04d02ae767b32e7ed7272: Deleting consensus metadata
I20260929 02:59:58.581517 1184 catalog_manager.cc:4841] TS c9a9a47ad9d04d02ae767b32e7ed7272 (127.15.18.67:33253): tablet fafb303405c24a8c85233fe90e0517a3 (table test-table [id=dcff7aff38cb4a459bab7fe152c55fb1]) successfully deleted
I20260929 02:59:58.669610 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 1261
I20260929 02:59:58.684523 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 1396
Sep 29 02:59:58 dist-test-slave-q5f0 krb5kdc[1155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 02:59:58.701853 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.685419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.15.18.66:36585)
0929 02:59:58.685581 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.685585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.685605 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:59:58.685751 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.685755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.685829 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.685937 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:58.687625 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.688353 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.690315 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.690777 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.690890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.696294 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:58.696312 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:58.696329 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:58.696375 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:58.699049 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:58.699949 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:58.699957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:58.699960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:58.700092 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:58.700470 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:58.700473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:58.700475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:58.700831 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:58.701066 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.701403 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.701573 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:59:58.702544 1520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:58.790632 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 1396
I20260929 02:59:58.805246 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 1530
Sep 29 02:59:58 dist-test-slave-q5f0 krb5kdc[1155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 02:59:58.824599 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:58.806154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.15.18.67:33253)
0929 02:59:58.806317 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:58.806321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:58.806341 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:59:58.806600 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:58.806609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:58.806691 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:58.806789 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:58.808687 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.809563 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.811476 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:58.812008 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:58.812113 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:58.817889 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:58.817905 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:58.817921 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:59:58.817970 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:58.821361 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:58.822409 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:58.822417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:58.822419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:58.822587 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:59:58.823054 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:59:58.823057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:59:58.823060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:59:58.823438 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:58.823708 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:58.824095 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 02:59:58.824280 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:59:58.825273 1654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:58.965822 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 1530
I20260929 02:59:58.980991 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 1168
I20260929 02:59:58.982131 1230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:59:59.094451 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 1168
2026-09-29T02:59:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4399 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T02:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:59:59Z Disabled control of system clock
I20260929 02:59:59.166482 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:38457
--webserver_interface=127.15.18.126
--webserver_port=0
--builtin_ntp_servers=127.15.18.84:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:38457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:59:59.501295 1712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:59:59.501605 1712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:59:59.501708 1712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:59:59.509210 1712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:59:59.509325 1712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:59:59.509384 1712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:59:59.509493 1712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:59:59.509564 1712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:59:59.519155 1712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:38457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:38457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 02:59:59.520460 1712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:59:59.522235 1712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:59:59.534119 1720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:59.534166 1718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:59:59.534375 1717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:59:59.534843 1712 server_base.cc:1034] running on GCE node
I20260929 02:59:59.535794 1712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:59:59.537299 1712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:59:59.538542 1712 hybrid_clock.cc:648] HybridClock initialized: now 1790650799538490 us; error 45 us; skew 500 ppm
I20260929 02:59:59.538913 1712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:59:59.541234 1712 webserver.cc:460] Webserver started at http://127.15.18.126:35047/ using document root <none> and password file <none>
I20260929 02:59:59.541941 1712 fs_manager.cc:359] Metadata directory not provided
I20260929 02:59:59.542040 1712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:59:59.542356 1712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:59:59.545265 1712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "d571b28594734ff1a515e2f906a04e22"
format_stamp: "Formatted at 2026-09-29 02:59:59 on dist-test-slave-q5f0"
server_key: "c7da85b21daa107a3989ab83fbba0d59"
server_key_iv: "7c588544cebe65ef71e683d299bb3375"
server_key_version: "encryptionkey@0"
I20260929 02:59:59.546057 1712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "d571b28594734ff1a515e2f906a04e22"
format_stamp: "Formatted at 2026-09-29 02:59:59 on dist-test-slave-q5f0"
server_key: "c7da85b21daa107a3989ab83fbba0d59"
server_key_iv: "7c588544cebe65ef71e683d299bb3375"
server_key_version: "encryptionkey@0"
I20260929 02:59:59.551196 1712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:59:59.554999 1726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:59:59.556429 1712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:59:59.556598 1712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "d571b28594734ff1a515e2f906a04e22"
format_stamp: "Formatted at 2026-09-29 02:59:59 on dist-test-slave-q5f0"
server_key: "c7da85b21daa107a3989ab83fbba0d59"
server_key_iv: "7c588544cebe65ef71e683d299bb3375"
server_key_version: "encryptionkey@0"
I20260929 02:59:59.556798 1712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:59:59.574952 1712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:59:59.575827 1712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:59:59.576104 1712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:59:59.593627 1712 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:38457
I20260929 02:59:59.593665 1777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:38457 every 8 connection(s)
I20260929 02:59:59.594964 1712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 02:59:59.598810 1778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:59:59.600464 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 1712
I20260929 02:59:59.600634 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 02:59:59.601169 15433 external_mini_cluster.cc:1428] Setting key edf0af9837803a5013a381a9d1902773
I20260929 02:59:59.611691 1778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22: Bootstrap starting.
I20260929 02:59:59.614173 1781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:59:59.603138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.15.18.126:38457)
0929 02:59:59.604204 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0929 02:59:59.604215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:59:59.604255 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:59:59.605826 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:59:59.605847 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:59:59.606321 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:59:59.606717 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:59:59.607236 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:59.608510 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:59.609921 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:59:59.610571 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:59:59.610739 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:59:59.611927 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:59:59.611939 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:59:59.611954 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:59:59.612482 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:59:59.612558 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:59:59.612582 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:59:59.612678 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 02:59:59.612920 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 02:59:59.613246 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":407,"threads_started":1}
I20260929 02:59:59.615082 1778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22: Neither blocks nor log segments found. Creating new log.
I20260929 02:59:59.616233 1778 log.cc:824] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22: Log is configured to *not* fsync() on all Append() calls
I20260929 02:59:59.619149 1778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22: No bootstrap required, opened a new log
I20260929 02:59:59.626478 1778 raft_consensus.cc:348] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571b28594734ff1a515e2f906a04e22" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38457 } }
I20260929 02:59:59.626858 1778 raft_consensus.cc:374] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:59:59.626956 1778 raft_consensus.cc:729] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d571b28594734ff1a515e2f906a04e22, State: Initialized, Role: FOLLOWER
I20260929 02:59:59.627578 1778 consensus_queue.cc:260] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571b28594734ff1a515e2f906a04e22" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38457 } }
I20260929 02:59:59.627810 1778 raft_consensus.cc:388] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:59:59.627936 1778 raft_consensus.cc:482] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:59:59.628085 1778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:59:59.630563 1778 raft_consensus.cc:504] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571b28594734ff1a515e2f906a04e22" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38457 } }
I20260929 02:59:59.631078 1778 leader_election.cc:302] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d571b28594734ff1a515e2f906a04e22; no voters:
I20260929 02:59:59.631731 1778 leader_election.cc:288] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:59:59.631886 1783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:59:59.632560 1783 raft_consensus.cc:686] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [term 1 LEADER]: Becoming Leader. State: Replica: d571b28594734ff1a515e2f906a04e22, State: Running, Role: LEADER
I20260929 02:59:59.633077 1783 consensus_queue.cc:237] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571b28594734ff1a515e2f906a04e22" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38457 } }
I20260929 02:59:59.633770 1778 sys_catalog.cc:564] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:59:59.635931 1784 sys_catalog.cc:455] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d571b28594734ff1a515e2f906a04e22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571b28594734ff1a515e2f906a04e22" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38457 } } }
I20260929 02:59:59.636111 1785 sys_catalog.cc:455] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d571b28594734ff1a515e2f906a04e22. Latest consensus state: current_term: 1 leader_uuid: "d571b28594734ff1a515e2f906a04e22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571b28594734ff1a515e2f906a04e22" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 38457 } } }
I20260929 02:59:59.636385 1784 sys_catalog.cc:458] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:59.636416 1785 sys_catalog.cc:458] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22 [sys.catalog]: This master's current role is: LEADER
I20260929 02:59:59.638504 1792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:59:59.643932 1792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:59:59.652655 1792 catalog_manager.cc:1347] Generated new cluster ID: e2bcd1fd5dd94dccaf738c1e45827004
I20260929 02:59:59.652781 1792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:59:59.663091 1792 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:59:59.664865 1792 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:59:59.675086 1792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22: Generated new TSK 0
I20260929 02:59:59.676054 1792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:59:59.690958 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38457
--builtin_ntp_servers=127.15.18.84:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:00:00.044072 1802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:00.044366 1802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:00.044494 1802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:00.051832 1802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:00.051942 1802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:00.052098 1802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 03:00:00.061470 1802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:00.062737 1802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:00.064558 1802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:00.077792 1808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:00.078109 1802 server_base.cc:1034] running on GCE node
W20260929 03:00:00.077761 1810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:00.077731 1807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:00.078958 1802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:00.079993 1802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:00.081204 1802 hybrid_clock.cc:648] HybridClock initialized: now 1790650800081144 us; error 55 us; skew 500 ppm
I20260929 03:00:00.081516 1802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:00:00.083814 1802 webserver.cc:460] Webserver started at http://127.15.18.65:35071/ using document root <none> and password file <none>
I20260929 03:00:00.084520 1802 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:00.084623 1802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:00.084939 1802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:00.088517 1802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "333d8b9e9c4a478883a430ce26cfb58a"
format_stamp: "Formatted at 2026-09-29 03:00:00 on dist-test-slave-q5f0"
server_key: "4702f1aed840bdf7d273dc59a3b2163f"
server_key_iv: "c7fd607b240514bde0f9265491716931"
server_key_version: "encryptionkey@0"
I20260929 03:00:00.089375 1802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "333d8b9e9c4a478883a430ce26cfb58a"
format_stamp: "Formatted at 2026-09-29 03:00:00 on dist-test-slave-q5f0"
server_key: "4702f1aed840bdf7d273dc59a3b2163f"
server_key_iv: "c7fd607b240514bde0f9265491716931"
server_key_version: "encryptionkey@0"
I20260929 03:00:00.095310 1802 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 03:00:00.099138 1816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:00.100621 1802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:00:00.100773 1802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "333d8b9e9c4a478883a430ce26cfb58a"
format_stamp: "Formatted at 2026-09-29 03:00:00 on dist-test-slave-q5f0"
server_key: "4702f1aed840bdf7d273dc59a3b2163f"
server_key_iv: "c7fd607b240514bde0f9265491716931"
server_key_version: "encryptionkey@0"
I20260929 03:00:00.100975 1802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:00.113912 1802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:00.114765 1802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:00.115026 1802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:00.116032 1802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:00:00.117486 1802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:00:00.117571 1802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:00.117663 1802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:00:00.117740 1802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:00.146076 1802 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:45649
I20260929 03:00:00.146124 1929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:45649 every 8 connection(s)
I20260929 03:00:00.147356 1802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 03:00:00.149791 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 1802
I20260929 03:00:00.149966 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 03:00:00.150465 15433 external_mini_cluster.cc:1428] Setting key 6d28db84f26a97ddf859f67389983c15
I20260929 03:00:00.153640 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38457
--builtin_ntp_servers=127.15.18.84:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:00:00.162070 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:00.150001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:38053 (local address 127.15.18.126:38457)
0929 03:00:00.150510 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:00.150516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:00.151843 (+ 1327us) server_negotiation.cc:407] Connection header received
0929 03:00:00.154201 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:00.154206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:00.154291 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:00.154389 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:00.155933 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:00.156694 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:00.158920 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:00.159349 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:00.159455 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:00.161198 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:00.161203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:00.161207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:00.161253 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:00.161307 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:00.161325 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:00.161432 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:00.161696 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 03:00:00.161788 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":238,"threads_started":1}
I20260929 03:00:00.163200 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:00.150632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38457 (local address 127.15.18.65:38053)
0929 03:00:00.151702 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:00.151738 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:00.153876 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:00.154698 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:00.154712 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:00.154847 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:00.155728 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:00.155754 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:00.156864 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:00.156869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:00.158729 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:00.158747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:00.159637 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:00.159644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:00.159945 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:00.160783 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:00:00.160786 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:00.160844 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:00:00.160956 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:00.161439 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:00.161451 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:00.161460 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:00:00.161664 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 03:00:00.162113 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":316,"threads_started":1}
I20260929 03:00:00.164898 1930 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38457
I20260929 03:00:00.165205 1930 heartbeater.cc:464] Registering TS with master...
I20260929 03:00:00.165977 1930 heartbeater.cc:511] Master 127.15.18.126:38457 requested a full tablet report, sending...
I20260929 03:00:00.168289 1743 ts_manager.cc:194] Registered new tserver with Master: 333d8b9e9c4a478883a430ce26cfb58a (127.15.18.65:45649)
I20260929 03:00:00.170286 1743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.65:38053
W20260929 03:00:00.482328 1934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:00.482636 1934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:00.482726 1934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:00.490113 1934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:00.490221 1934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:00.490330 1934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 03:00:00.499600 1934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:00.500963 1934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:00.502667 1934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:00.515436 1939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:00.515465 1940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:00.515491 1943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:00.516050 1934 server_base.cc:1034] running on GCE node
I20260929 03:00:00.516669 1934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:00.517647 1934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:00.518862 1934 hybrid_clock.cc:648] HybridClock initialized: now 1790650800518797 us; error 61 us; skew 500 ppm
I20260929 03:00:00.519179 1934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:00:00.521452 1934 webserver.cc:460] Webserver started at http://127.15.18.66:42657/ using document root <none> and password file <none>
I20260929 03:00:00.522157 1934 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:00.522256 1934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:00.522562 1934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:00.525492 1934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "60aa18aebe8a428c83c91f61b211dc90"
format_stamp: "Formatted at 2026-09-29 03:00:00 on dist-test-slave-q5f0"
server_key: "5f0c60dadd842a958391fe398441e3d5"
server_key_iv: "ec787035e33d3126d9484ef3e7de9745"
server_key_version: "encryptionkey@0"
I20260929 03:00:00.526288 1934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "60aa18aebe8a428c83c91f61b211dc90"
format_stamp: "Formatted at 2026-09-29 03:00:00 on dist-test-slave-q5f0"
server_key: "5f0c60dadd842a958391fe398441e3d5"
server_key_iv: "ec787035e33d3126d9484ef3e7de9745"
server_key_version: "encryptionkey@0"
I20260929 03:00:00.531786 1934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 03:00:00.535359 1949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:00.536851 1934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:00:00.537020 1934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "60aa18aebe8a428c83c91f61b211dc90"
format_stamp: "Formatted at 2026-09-29 03:00:00 on dist-test-slave-q5f0"
server_key: "5f0c60dadd842a958391fe398441e3d5"
server_key_iv: "ec787035e33d3126d9484ef3e7de9745"
server_key_version: "encryptionkey@0"
I20260929 03:00:00.537225 1934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:00.570168 1934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:00.571025 1934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:00.571321 1934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:00.572358 1934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:00:00.573930 1934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:00:00.574028 1934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:00.574137 1934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:00:00.574204 1934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:00.608837 1934 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:36015
I20260929 03:00:00.608871 2061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:36015 every 8 connection(s)
I20260929 03:00:00.610145 1934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 03:00:00.612407 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 1934
I20260929 03:00:00.612601 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 03:00:00.613140 15433 external_mini_cluster.cc:1428] Setting key 75264af0f7ae00bfa9bbd413ae6bc9ff
I20260929 03:00:00.615823 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38457
--builtin_ntp_servers=127.15.18.84:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:00:00.623081 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:00.612661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:37377 (local address 127.15.18.126:38457)
0929 03:00:00.612906 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:00.612913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:00.614383 (+ 1470us) server_negotiation.cc:407] Connection header received
0929 03:00:00.616215 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:00.616221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:00.616303 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:00.616387 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:00.617514 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:00.618211 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:00.620218 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:00.620626 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:00.620742 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:00.622160 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:00.622167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:00.622172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:00.622198 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:00.622243 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:00.622260 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:00.622359 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:00.622635 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 03:00:00.622750 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:00:00.624224 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:00.613181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38457 (local address 127.15.18.66:37377)
0929 03:00:00.614203 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:00.614257 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:00.615933 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:00.616653 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:00.616661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:00.616759 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:00.617343 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:00.617360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:00.618364 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:00.618367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:00.620042 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:00.620057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:00.620876 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:00.620884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:00.621101 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:00.621795 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:00:00.621797 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:00.621840 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:00:00.621927 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:00.622398 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:00.622409 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:00.622416 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:00:00.622607 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 03:00:00.622957 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":288,"threads_started":1}
I20260929 03:00:00.625808 2062 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38457
I20260929 03:00:00.626112 2062 heartbeater.cc:464] Registering TS with master...
I20260929 03:00:00.626880 2062 heartbeater.cc:511] Master 127.15.18.126:38457 requested a full tablet report, sending...
I20260929 03:00:00.628386 1743 ts_manager.cc:194] Registered new tserver with Master: 60aa18aebe8a428c83c91f61b211dc90 (127.15.18.66:36015)
I20260929 03:00:00.629212 1743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.66:37377
W20260929 03:00:00.936540 2065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:00.936823 2065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:00.936925 2065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:00.944351 2065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:00.944458 2065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:00.944589 2065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 03:00:00.953814 2065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:38457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:00.955097 2065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:00.956843 2065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:00.969470 2070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:00.969434 2071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:00.969609 2073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:00.970275 2065 server_base.cc:1034] running on GCE node
I20260929 03:00:00.970984 2065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:00.972015 2065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:00.973278 2065 hybrid_clock.cc:648] HybridClock initialized: now 1790650800973224 us; error 39 us; skew 500 ppm
I20260929 03:00:00.973626 2065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:00:00.976089 2065 webserver.cc:460] Webserver started at http://127.15.18.67:45679/ using document root <none> and password file <none>
I20260929 03:00:00.976828 2065 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:00.976965 2065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:00.977298 2065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:00.980340 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "ab063e873e3c42e1b0b8a9375ae11e2d"
format_stamp: "Formatted at 2026-09-29 03:00:00 on dist-test-slave-q5f0"
server_key: "beaaf0f61c48867cb68fe5973fe90493"
server_key_iv: "4fd13919f33a784a768dea13d2f73889"
server_key_version: "encryptionkey@0"
I20260929 03:00:00.981169 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "ab063e873e3c42e1b0b8a9375ae11e2d"
format_stamp: "Formatted at 2026-09-29 03:00:00 on dist-test-slave-q5f0"
server_key: "beaaf0f61c48867cb68fe5973fe90493"
server_key_iv: "4fd13919f33a784a768dea13d2f73889"
server_key_version: "encryptionkey@0"
I20260929 03:00:00.986384 2065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:00:00.990098 2080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:00.991463 2065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:00:00.991696 2065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "ab063e873e3c42e1b0b8a9375ae11e2d"
format_stamp: "Formatted at 2026-09-29 03:00:00 on dist-test-slave-q5f0"
server_key: "beaaf0f61c48867cb68fe5973fe90493"
server_key_iv: "4fd13919f33a784a768dea13d2f73889"
server_key_version: "encryptionkey@0"
I20260929 03:00:00.991912 2065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:01.008117 2065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:01.009033 2065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:01.009356 2065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:01.010545 2065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:00:01.012080 2065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:00:01.012179 2065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:01.012295 2065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:00:01.012373 2065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:01.043591 2065 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:37859
I20260929 03:00:01.043653 2192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:37859 every 8 connection(s)
I20260929 03:00:01.045004 2065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 03:00:01.050206 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 2065
I20260929 03:00:01.050374 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 03:00:01.050777 15433 external_mini_cluster.cc:1428] Setting key 9480dadc3662ac569ca5cfbd15c32eb9
I20260929 03:00:01.057271 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.047406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:57823 (local address 127.15.18.126:38457)
0929 03:00:01.047606 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:01.047610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:01.048976 (+ 1366us) server_negotiation.cc:407] Connection header received
0929 03:00:01.050764 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:01.050769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:01.050850 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:01.050952 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:01.052113 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.052946 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.054700 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.055125 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.055240 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.056580 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:01.056585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:01.056589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:01.056611 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:01.056644 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:01.056657 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:01.056744 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:01.056923 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 03:00:01.057010 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:00:01.058218 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.047940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:38457 (local address 127.15.18.67:57823)
0929 03:00:01.048795 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.048836 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:01.050502 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:01.051223 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:01.051235 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:01.051334 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:01.051974 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.051992 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.053097 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.053102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.054542 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.054554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.055328 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.055333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.055608 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.056266 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:00:01.056269 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:01.056309 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:00:01.056395 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:01.056742 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:01.056748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:01.056754 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:00:01.056885 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 03:00:01.057184 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":256,"threads_started":1}
I20260929 03:00:01.059685 2193 heartbeater.cc:347] Connected to a master server at 127.15.18.126:38457
I20260929 03:00:01.059975 2193 heartbeater.cc:464] Registering TS with master...
I20260929 03:00:01.060768 2193 heartbeater.cc:511] Master 127.15.18.126:38457 requested a full tablet report, sending...
I20260929 03:00:01.062137 1743 ts_manager.cc:194] Registered new tserver with Master: ab063e873e3c42e1b0b8a9375ae11e2d (127.15.18.67:37859)
I20260929 03:00:01.062915 1743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.67:57823
I20260929 03:00:01.066087 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:00:01.076061 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.070496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.15.18.126:38457)
0929 03:00:01.070698 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:01.070703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:01.071027 (+ 324us) server_negotiation.cc:407] Connection header received
0929 03:00:01.071181 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:01.071185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:01.071261 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:01.071363 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:01.071899 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.072605 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.073881 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.074287 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.074383 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.075382 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:01.075386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:01.075390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:01.075407 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:01.075445 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:01.075457 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:01.075623 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:01.075764 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 03:00:01.075813 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:00:01.080024 1743 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:00:01.086764 1743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:00:01.128872 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.099364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:36015 (local address 127.0.0.1:45560)
0929 03:00:01.119309 (+ 19945us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.119326 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:01.119425 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:01.120814 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:01.120822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:01.120855 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:01.121265 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.121278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.123167 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.123172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.125642 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.125656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.127897 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.127904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.128116 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.128170 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 03:00:01.128314 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 03:00:01.128525 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19834,"thread_start_us":20143,"threads_started":1}
I20260929 03:00:01.128903 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.100182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.15.18.66:36015)
0929 03:00:01.120135 (+ 19953us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:01.120144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:01.120177 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:00:01.120286 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:01.120294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:01.120577 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:01.120784 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:01.121474 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.123044 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.125851 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.127710 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.127843 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.128105 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:01.128394 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 03:00:01.128570 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19759,"thread_start_us":19644,"threads_started":1}
I20260929 03:00:01.130031 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.098947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:37859 (local address 127.0.0.1:49564)
0929 03:00:01.114581 (+ 15634us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.114625 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:01.114847 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:01.123405 (+ 8558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:01.123412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:01.123437 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:01.123897 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.123909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.125850 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.125855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.127482 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.127498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.129299 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.129305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.129504 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.129536 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:00:01.129609 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:00:01.129738 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15332,"thread_start_us":15177,"threads_started":1}
I20260929 03:00:01.130224 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.099207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.15.18.67:37859)
0929 03:00:01.122577 (+ 23370us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:01.122584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:01.122613 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:00:01.122701 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:01.122721 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:01.122945 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:01.123130 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:01.124115 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.125607 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.127707 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.129059 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.129181 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.129491 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:01.129699 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 03:00:01.129885 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23194,"thread_start_us":23031,"threads_started":1}
I20260929 03:00:01.129762 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.099044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:45649 (local address 127.0.0.1:60264)
0929 03:00:01.118686 (+ 19642us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.118719 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:01.118864 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:01.121930 (+ 3066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:01.121934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:01.121957 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:01.122316 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.122328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.124588 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.124592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.126442 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.126457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.128473 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.128479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.128641 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.128675 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 03:00:01.128768 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:00:01.128898 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19460,"mutex_wait_us":1,"thread_start_us":20045,"threads_started":1}
I20260929 03:00:01.130934 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.099721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.15.18.65:45649)
0929 03:00:01.121435 (+ 21714us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:01.121443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:01.121471 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:00:01.121548 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:01.121553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:01.121740 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:01.121902 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:01.122766 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.124450 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.126665 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.128044 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.128167 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.128345 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:01.129850 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0929 03:00:01.130628 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21547,"thread_start_us":21418,"threads_started":1}
I20260929 03:00:01.133481 1997 tablet_service.cc:1467] Processing CreateTablet for tablet fddd3eb55c804a2987d400eb12bb5772 (DEFAULT_TABLE table=test-table [id=c2e741247897498ea86754debeb4737a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:00:01.135067 1997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddd3eb55c804a2987d400eb12bb5772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:01.135164 1865 tablet_service.cc:1467] Processing CreateTablet for tablet fddd3eb55c804a2987d400eb12bb5772 (DEFAULT_TABLE table=test-table [id=c2e741247897498ea86754debeb4737a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:00:01.135462 2128 tablet_service.cc:1467] Processing CreateTablet for tablet fddd3eb55c804a2987d400eb12bb5772 (DEFAULT_TABLE table=test-table [id=c2e741247897498ea86754debeb4737a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:00:01.136936 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddd3eb55c804a2987d400eb12bb5772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:01.136997 1865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddd3eb55c804a2987d400eb12bb5772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:01.153165 2213 tablet_bootstrap.cc:492] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: Bootstrap starting.
I20260929 03:00:01.158239 2214 tablet_bootstrap.cc:492] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a: Bootstrap starting.
I20260929 03:00:01.159320 2213 tablet_bootstrap.cc:654] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:01.160737 2213 log.cc:824] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:01.162124 2212 tablet_bootstrap.cc:492] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: Bootstrap starting.
I20260929 03:00:01.164242 2214 tablet_bootstrap.cc:654] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:01.165596 2214 log.cc:824] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:01.165901 2213 tablet_bootstrap.cc:492] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: No bootstrap required, opened a new log
I20260929 03:00:01.165911 2212 tablet_bootstrap.cc:654] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:01.166195 2213 ts_tablet_manager.cc:1397] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20260929 03:00:01.166988 2212 log.cc:824] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:01.169060 2214 tablet_bootstrap.cc:492] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a: No bootstrap required, opened a new log
I20260929 03:00:01.169443 2214 ts_tablet_manager.cc:1397] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 03:00:01.170404 2212 tablet_bootstrap.cc:492] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: No bootstrap required, opened a new log
I20260929 03:00:01.170688 2212 ts_tablet_manager.cc:1397] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260929 03:00:01.173574 1930 heartbeater.cc:503] Master 127.15.18.126:38457 was elected leader, sending a full tablet report...
I20260929 03:00:01.177608 2213 raft_consensus.cc:348] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 37859 } } peers { permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 } } peers { permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 } }
I20260929 03:00:01.178293 2213 raft_consensus.cc:374] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:01.178457 2213 raft_consensus.cc:729] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60aa18aebe8a428c83c91f61b211dc90, State: Initialized, Role: FOLLOWER
I20260929 03:00:01.178197 2214 raft_consensus.cc:348] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 37859 } } peers { permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 } } peers { permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 } }
I20260929 03:00:01.178778 2214 raft_consensus.cc:374] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:01.178912 2214 raft_consensus.cc:729] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333d8b9e9c4a478883a430ce26cfb58a, State: Initialized, Role: FOLLOWER
I20260929 03:00:01.179628 2213 consensus_queue.cc:260] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 37859 } } peers { permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 } } peers { permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 } }
I20260929 03:00:01.179692 2214 consensus_queue.cc:260] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 37859 } } peers { permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 } } peers { permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 } }
I20260929 03:00:01.181103 2212 raft_consensus.cc:348] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 37859 } } peers { permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 } } peers { permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 } }
I20260929 03:00:01.181617 2212 raft_consensus.cc:374] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:01.181762 2212 raft_consensus.cc:729] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab063e873e3c42e1b0b8a9375ae11e2d, State: Initialized, Role: FOLLOWER
I20260929 03:00:01.182626 2212 consensus_queue.cc:260] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 37859 } } peers { permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 } } peers { permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 } }
I20260929 03:00:01.183203 2214 ts_tablet_manager.cc:1428] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260929 03:00:01.185021 2062 heartbeater.cc:503] Master 127.15.18.126:38457 was elected leader, sending a full tablet report...
I20260929 03:00:01.185887 2213 ts_tablet_manager.cc:1428] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: Time spent starting tablet: real 0.020s user 0.014s sys 0.003s
I20260929 03:00:01.187134 2193 heartbeater.cc:503] Master 127.15.18.126:38457 was elected leader, sending a full tablet report...
I20260929 03:00:01.187925 2212 ts_tablet_manager.cc:1428] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: Time spent starting tablet: real 0.017s user 0.019s sys 0.000s
W20260929 03:00:01.296697 2194 tablet.cc:2369] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:00:01.362121 2063 tablet.cc:2369] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:00:01.399892 1931 tablet.cc:2369] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:00:01.481638 2220 raft_consensus.cc:482] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:00:01.481976 2220 raft_consensus.cc:504] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 37859 } } peers { permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 } } peers { permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 } }
I20260929 03:00:01.483853 2220 leader_election.cc:288] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 333d8b9e9c4a478883a430ce26cfb58a (127.15.18.65:45649), 60aa18aebe8a428c83c91f61b211dc90 (127.15.18.66:36015)
I20260929 03:00:01.489593 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.484324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:52243 (local address 127.15.18.66:36015)
0929 03:00:01.484538 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:01.484545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:01.484614 (+ 69us) server_negotiation.cc:407] Connection header received
0929 03:00:01.484856 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:01.484861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:01.484936 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:01.485012 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:01.485464 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.486208 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.487924 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.488686 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.488845 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.488921 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:01.489170 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 03:00:01.489292 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 03:00:01.489624 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.484191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:36015 (local address 127.15.18.67:52243)
0929 03:00:01.484527 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.484551 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:01.484696 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:01.485114 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:01.485119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:01.485146 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:01.485301 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.485309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.486348 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.486354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.487742 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.487756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.488846 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.488852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.489015 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.489038 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:00:01.489116 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:00:01.489284 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260929 03:00:01.490242 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.484230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:45649 (local address 127.15.18.67:39565)
0929 03:00:01.484957 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.484972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:01.485106 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:01.485446 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:01.485450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:01.485472 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:01.485790 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.485798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.486809 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.486813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.488234 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:01.488247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.489571 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.489577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:01.489733 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.489770 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:00:01.489892 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 03:00:01.490015 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":396,"threads_started":1}
I20260929 03:00:01.490340 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.484422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:39565 (local address 127.15.18.65:45649)
0929 03:00:01.484640 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:01.484647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:01.485078 (+ 431us) server_negotiation.cc:407] Connection header received
0929 03:00:01.485250 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:01.485254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:01.485328 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:01.485402 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:01.485955 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.486651 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.488426 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.489205 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.489357 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.489441 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:01.489962 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 03:00:01.490106 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:00:01.490767 2017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddd3eb55c804a2987d400eb12bb5772" candidate_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60aa18aebe8a428c83c91f61b211dc90" is_pre_election: true
I20260929 03:00:01.491351 2017 raft_consensus.cc:2454] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab063e873e3c42e1b0b8a9375ae11e2d in term 0.
I20260929 03:00:01.491371 1885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddd3eb55c804a2987d400eb12bb5772" candidate_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333d8b9e9c4a478883a430ce26cfb58a" is_pre_election: true
I20260929 03:00:01.491922 1885 raft_consensus.cc:2454] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab063e873e3c42e1b0b8a9375ae11e2d in term 0.
I20260929 03:00:01.492127 2084 leader_election.cc:302] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60aa18aebe8a428c83c91f61b211dc90, ab063e873e3c42e1b0b8a9375ae11e2d; no voters:
I20260929 03:00:01.492530 2220 raft_consensus.cc:2781] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:00:01.492682 2220 raft_consensus.cc:482] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:00:01.492779 2220 raft_consensus.cc:3037] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:01.494831 2220 raft_consensus.cc:504] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 37859 } } peers { permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 } } peers { permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 } }
I20260929 03:00:01.495414 2220 leader_election.cc:288] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [CANDIDATE]: Term 1 election: Requested vote from peers 333d8b9e9c4a478883a430ce26cfb58a (127.15.18.65:45649), 60aa18aebe8a428c83c91f61b211dc90 (127.15.18.66:36015)
I20260929 03:00:01.495858 1885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddd3eb55c804a2987d400eb12bb5772" candidate_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333d8b9e9c4a478883a430ce26cfb58a"
I20260929 03:00:01.495944 2017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddd3eb55c804a2987d400eb12bb5772" candidate_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60aa18aebe8a428c83c91f61b211dc90"
I20260929 03:00:01.496059 1885 raft_consensus.cc:3037] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:01.496160 2017 raft_consensus.cc:3037] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:01.498521 1885 raft_consensus.cc:2454] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab063e873e3c42e1b0b8a9375ae11e2d in term 1.
I20260929 03:00:01.498572 2017 raft_consensus.cc:2454] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab063e873e3c42e1b0b8a9375ae11e2d in term 1.
I20260929 03:00:01.498915 2081 leader_election.cc:302] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333d8b9e9c4a478883a430ce26cfb58a, ab063e873e3c42e1b0b8a9375ae11e2d; no voters:
I20260929 03:00:01.499215 2220 raft_consensus.cc:2781] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:01.499635 2220 raft_consensus.cc:686] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 1 LEADER]: Becoming Leader. State: Replica: ab063e873e3c42e1b0b8a9375ae11e2d, State: Running, Role: LEADER
I20260929 03:00:01.500214 2220 consensus_queue.cc:237] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 37859 } } peers { permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 } } peers { permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 } }
I20260929 03:00:01.504957 1742 catalog_manager.cc:5495] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d reported cstate change: term changed from 0 to 1, leader changed from <none> to ab063e873e3c42e1b0b8a9375ae11e2d (127.15.18.67). New cstate: current_term: 1 leader_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab063e873e3c42e1b0b8a9375ae11e2d" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 } health_report { overall_health: UNKNOWN } } }
I20260929 03:00:01.599656 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.593068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49574 (local address 127.15.18.67:37859)
0929 03:00:01.593274 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:01.593280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:01.593446 (+ 166us) server_negotiation.cc:407] Connection header received
0929 03:00:01.593642 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:01.593648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:01.593775 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:01.593928 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:01.594304 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.595424 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.596992 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.597639 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.597766 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.598365 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:01.598997 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:01.599160 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:01.599316 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:00:01.599362 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:00:01.608508 1885 raft_consensus.cc:1261] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a [term 1 FOLLOWER]: Refusing update from remote peer ab063e873e3c42e1b0b8a9375ae11e2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:00:01.608520 2017 raft_consensus.cc:1261] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90 [term 1 FOLLOWER]: Refusing update from remote peer ab063e873e3c42e1b0b8a9375ae11e2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:00:01.609664 2222 consensus_queue.cc:1035] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60aa18aebe8a428c83c91f61b211dc90" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 36015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:00:01.610117 2220 consensus_queue.cc:1035] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "333d8b9e9c4a478883a430ce26cfb58a" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:00:01.625434 2226 mvcc.cc:205] Tried to move back new op lower bound from 7334505683377819648 to 7334505682948960256. Current Snapshot: MvccSnapshot[applied={T|T < 7334505683377819648 or (T in {7334505683377819648})}]
I20260929 03:00:01.643137 1742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:00:01.643441 1742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:00:01.647357 1742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d571b28594734ff1a515e2f906a04e22: Sending DeleteTablet for 3 replicas of tablet fddd3eb55c804a2987d400eb12bb5772
I20260929 03:00:01.648619 2128 tablet_service.cc:1514] Processing DeleteTablet for tablet fddd3eb55c804a2987d400eb12bb5772 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:00:01 UTC) from {username='slave'} at 127.0.0.1:49564
I20260929 03:00:01.648788 1997 tablet_service.cc:1514] Processing DeleteTablet for tablet fddd3eb55c804a2987d400eb12bb5772 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:00:01 UTC) from {username='slave'} at 127.0.0.1:45560
I20260929 03:00:01.648766 1865 tablet_service.cc:1514] Processing DeleteTablet for tablet fddd3eb55c804a2987d400eb12bb5772 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:00:01 UTC) from {username='slave'} at 127.0.0.1:60264
I20260929 03:00:01.650839 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 1802
I20260929 03:00:01.682780 2236 tablet_replica.cc:330] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: stopping tablet replica
I20260929 03:00:01.683447 2236 raft_consensus.cc:2229] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:00:01.684232 2236 raft_consensus.cc:2258] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:00:01.688939 2236 ts_tablet_manager.cc:1905] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:00:01.691907 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.652301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.15.18.65:45649)
0929 03:00:01.652495 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:01.652501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:01.677350 (+ 24849us) server_negotiation.cc:407] Connection header received
0929 03:00:01.677916 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:01.677924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:01.678060 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:01.678142 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:01.678942 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.680239 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.684586 (+ 4347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.685363 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.685475 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.689791 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:01.689805 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:01.689826 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:01.690746 (+ 920us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:01.690824 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:01.690857 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:01.690961 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:01.691266 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 03:00:01.691437 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:00:01.693598 1925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:01.696238 2235 tablet_replica.cc:330] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: stopping tablet replica
I20260929 03:00:01.696856 2235 raft_consensus.cc:2229] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:00:01.697619 2235 raft_consensus.cc:2258] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:00:01.699712 2235 ts_tablet_manager.cc:1905] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:00:01.701905 2236 ts_tablet_manager.cc:1918] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:00:01.702633 2236 log.cc:1196] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/fddd3eb55c804a2987d400eb12bb5772
I20260929 03:00:01.704033 2236 ts_tablet_manager.cc:1939] T fddd3eb55c804a2987d400eb12bb5772 P 60aa18aebe8a428c83c91f61b211dc90: Deleting consensus metadata
I20260929 03:00:01.706113 1730 catalog_manager.cc:4841] TS 60aa18aebe8a428c83c91f61b211dc90 (127.15.18.66:36015): tablet fddd3eb55c804a2987d400eb12bb5772 (table test-table [id=c2e741247897498ea86754debeb4737a]) successfully deleted
I20260929 03:00:01.706667 2235 ts_tablet_manager.cc:1918] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:00:01.706810 2235 log.cc:1196] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/fddd3eb55c804a2987d400eb12bb5772
I20260929 03:00:01.707262 2235 ts_tablet_manager.cc:1939] T fddd3eb55c804a2987d400eb12bb5772 P ab063e873e3c42e1b0b8a9375ae11e2d: Deleting consensus metadata
I20260929 03:00:01.708724 1729 catalog_manager.cc:4841] TS ab063e873e3c42e1b0b8a9375ae11e2d (127.15.18.67:37859): tablet fddd3eb55c804a2987d400eb12bb5772 (table test-table [id=c2e741247897498ea86754debeb4737a]) successfully deleted
I20260929 03:00:01.807229 2237 tablet_replica.cc:330] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a: stopping tablet replica
I20260929 03:00:01.807617 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 1802
I20260929 03:00:01.807732 2237 raft_consensus.cc:2229] T fddd3eb55c804a2987d400eb12bb5772 P 333d8b9e9c4a478883a430ce26cfb58a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 03:00:01.829164 1727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.18.65:45649 (error 108)
I20260929 03:00:01.829949 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 1934
I20260929 03:00:01.831686 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.830598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:45649 (local address 127.0.0.1:60278)
0929 03:00:01.831251 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.831414 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":589,"threads_started":1}
W20260929 03:00:01.832134 1727 catalog_manager.cc:4584] TS 333d8b9e9c4a478883a430ce26cfb58a (127.15.18.65:45649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fddd3eb55c804a2987d400eb12bb5772: Network error: Client connection negotiation failed: client connection to 127.15.18.65:45649: connect: Connection refused (error 111)
I20260929 03:00:01.837507 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.830808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.15.18.66:36015)
0929 03:00:01.831015 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:01.831022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:01.831046 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:00:01.831236 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:01.831242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:01.831357 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:01.831469 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:01.831980 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.832881 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.834299 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:01.834718 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:01.834845 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:01.836060 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:01.836069 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:01.836083 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:01.836727 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:01.836774 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:01.836794 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:01.836897 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:01.837041 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 03:00:01.837153 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:00:01.838140 2057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:01.886103 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.885631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:45649 (local address 127.0.0.1:60290)
0929 03:00:01.885860 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.885952 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 03:00:01.887323 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.886934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:45649 (local address 127.0.0.1:60300)
0929 03:00:01.887102 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.887175 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 03:00:01.947882 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.947345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:45649 (local address 127.0.0.1:60312)
0929 03:00:01.947596 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.947686 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 03:00:01.948982 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:01.948613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:45649 (local address 127.0.0.1:60322)
0929 03:00:01.948771 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:01.948842 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 03:00:01.986178 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 1934
I20260929 03:00:02.001112 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 2065
I20260929 03:00:02.012825 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:02.001883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.15.18.67:37859)
0929 03:00:02.002078 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:02.002084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:02.002107 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:00:02.002266 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:02.002272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:02.002391 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:02.002506 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:02.002911 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:02.004029 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:02.005301 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:02.005945 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:02.006080 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:02.010955 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:02.010968 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:02.010990 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:02.011916 (+ 926us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:02.011987 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:02.012017 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:02.012144 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:02.012313 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 03:00:02.012430 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:00:02.013746 2188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:02.029906 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:02.029264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:45649 (local address 127.0.0.1:60336)
0929 03:00:02.029579 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:02.029698 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 03:00:02.031143 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:02.030679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:45649 (local address 127.0.0.1:60348)
0929 03:00:02.030900 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:02.030994 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 03:00:02.135149 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 2065
I20260929 03:00:02.150727 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 1712
I20260929 03:00:02.151919 1773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:02.262374 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 1712
I20260929 03:00:02.262480 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:02.262001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:45649 (local address 127.0.0.1:60354)
0929 03:00:02.262179 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:02.262294 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.65:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
2026-09-29T03:00:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3166 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:02 dist-test-slave-q5f0 krb5kdc[2247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:02 dist-test-slave-q5f0 krb5kdc[2247](info): set up 2 sockets
Sep 29 03:00:02 dist-test-slave-q5f0 krb5kdc[2247](info): commencing operation
krb5kdc: starting...
W20260929 03:00:03.292018 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:03 dist-test-slave-q5f0 krb5kdc[2247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 03:00:03.543149 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:45473
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:45473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:03.874771 2260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:03.875100 2260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:03.875223 2260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:03.882766 2260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:03.882927 2260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:03.882975 2260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:03.883231 2260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:03.883314 2260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:03.894004 2260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:45473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:45473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:03.895437 2260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:03.896848 2260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:03.908965 2268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:03.909106 2265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:03.909047 2266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:03.910145 2260 server_base.cc:1034] running on GCE node
I20260929 03:00:03.910974 2260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:03.912567 2260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:03.913820 2260 hybrid_clock.cc:648] HybridClock initialized: now 1790650803913780 us; error 37 us; skew 500 ppm
I20260929 03:00:03.914182 2260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:03 dist-test-slave-q5f0 krb5kdc[2247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650803, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:03.917366 2260 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 03:00:03.918970 2260 webserver.cc:460] Webserver started at http://127.15.18.126:40491/ using document root <none> and password file <none>
I20260929 03:00:03.919719 2260 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:03.919850 2260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:03.920143 2260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:03.923180 2260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "9272169e3da04c0487d6848ad587d496"
format_stamp: "Formatted at 2026-09-29 03:00:03 on dist-test-slave-q5f0"
server_key: "2a7bc783f2de11ecc48f8716ff8d46b4"
server_key_iv: "cb30b5a9ba4699840a882b5562588893"
server_key_version: "encryptionkey@0"
I20260929 03:00:03.923970 2260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "9272169e3da04c0487d6848ad587d496"
format_stamp: "Formatted at 2026-09-29 03:00:03 on dist-test-slave-q5f0"
server_key: "2a7bc783f2de11ecc48f8716ff8d46b4"
server_key_iv: "cb30b5a9ba4699840a882b5562588893"
server_key_version: "encryptionkey@0"
I20260929 03:00:03.929385 2260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:00:03.933195 2275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:03.934695 2260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:00:03.934906 2260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "9272169e3da04c0487d6848ad587d496"
format_stamp: "Formatted at 2026-09-29 03:00:03 on dist-test-slave-q5f0"
server_key: "2a7bc783f2de11ecc48f8716ff8d46b4"
server_key_iv: "cb30b5a9ba4699840a882b5562588893"
server_key_version: "encryptionkey@0"
I20260929 03:00:03.935109 2260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:03.955174 2260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:03.959956 2260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:03.960242 2260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:03.979130 2260 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:45473
I20260929 03:00:03.979148 2326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:45473 every 8 connection(s)
I20260929 03:00:03.980597 2260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:03.984438 2327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:03.987972 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 2260
I20260929 03:00:03.988163 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:03.988684 15433 external_mini_cluster.cc:1428] Setting key 0051eda9d8f43bc6eea5ad3cd5a76c9e
I20260929 03:00:03.997816 2327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496: Bootstrap starting.
I20260929 03:00:04.002156 2327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:04.003626 2327 log.cc:824] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:00:04 dist-test-slave-q5f0 krb5kdc[2247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 03:00:04.007007 2327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496: No bootstrap required, opened a new log
I20260929 03:00:04.014398 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:03.990733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.15.18.126:45473)
0929 03:00:03.991610 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:03.991623 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:03.991668 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:00:03.993526 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:03.993552 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:03.994015 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:03.994406 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:03.996308 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:03.997706 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:03.999947 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:04.000733 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:04.000878 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:04.006270 (+ 5392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:04.006279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:04.006295 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:04.006342 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:04.009813 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:04.010686 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:04.010694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:04.010701 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:04.010888 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:04.011328 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:04.011332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:04.011334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:04.011988 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:04.012307 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:04.012817 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 03:00:04.013269 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":379,"threads_started":1}
I20260929 03:00:04.015350 2327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9272169e3da04c0487d6848ad587d496" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45473 } }
I20260929 03:00:04.015846 2327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:04.015962 2327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9272169e3da04c0487d6848ad587d496, State: Initialized, Role: FOLLOWER
I20260929 03:00:04.016588 2327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9272169e3da04c0487d6848ad587d496" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45473 } }
I20260929 03:00:04.016819 2327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:04.016960 2327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:04.017117 2327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:04.019461 2327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9272169e3da04c0487d6848ad587d496" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45473 } }
I20260929 03:00:04.020023 2327 leader_election.cc:302] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9272169e3da04c0487d6848ad587d496; no voters:
I20260929 03:00:04.020653 2327 leader_election.cc:288] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:04.020763 2332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:04.021809 2332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [term 1 LEADER]: Becoming Leader. State: Replica: 9272169e3da04c0487d6848ad587d496, State: Running, Role: LEADER
I20260929 03:00:04.022341 2332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9272169e3da04c0487d6848ad587d496" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45473 } }
I20260929 03:00:04.022830 2327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:04.025125 2333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9272169e3da04c0487d6848ad587d496" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9272169e3da04c0487d6848ad587d496" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45473 } } }
I20260929 03:00:04.025566 2333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:04.025525 2334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9272169e3da04c0487d6848ad587d496. Latest consensus state: current_term: 1 leader_uuid: "9272169e3da04c0487d6848ad587d496" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9272169e3da04c0487d6848ad587d496" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 45473 } } }
I20260929 03:00:04.025969 2334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:04.028074 2339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:04.033640 2339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:04.043113 2339 catalog_manager.cc:1347] Generated new cluster ID: 4e8735dfb1dd44209e2ec0999c09344f
I20260929 03:00:04.043298 2339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:04.093657 2339 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:04.095691 2339 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:04.105607 2339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496: Generated new TSK 0
I20260929 03:00:04.106508 2339 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.18.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
Entry for principal kudu/127.15.18.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab.
I20260929 03:00:04.179539 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45473
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:04.521867 2353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:04.522168 2353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:04.522303 2353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:04.529876 2353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:04.530009 2353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:04.530169 2353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 03:00:04.540623 2353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.65.keytab
--principal=kudu/127.15.18.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:04.541970 2353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:04.543341 2353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:04.556649 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:04.556676 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:04.556805 2361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:04.557696 2353 server_base.cc:1034] running on GCE node
I20260929 03:00:04.558359 2353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:04.559430 2353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:04.560691 2353 hybrid_clock.cc:648] HybridClock initialized: now 1790650804560628 us; error 50 us; skew 500 ppm
I20260929 03:00:04.561054 2353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:04 dist-test-slave-q5f0 krb5kdc[2247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650804, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:04.564278 2353 init.cc:377] Logged in from keytab as kudu/127.15.18.65@KRBTEST.COM (short username kudu)
I20260929 03:00:04.565876 2353 webserver.cc:460] Webserver started at http://127.15.18.65:40987/ using document root <none> and password file <none>
I20260929 03:00:04.566552 2353 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:04.566653 2353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:04.566952 2353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:04.569988 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "0540b11e49f44e6ca238bd5a2c047f8e"
format_stamp: "Formatted at 2026-09-29 03:00:04 on dist-test-slave-q5f0"
server_key: "971625a8082c895514809073b2667f61"
server_key_iv: "f21b14204b451e717bf937fada589b43"
server_key_version: "encryptionkey@0"
I20260929 03:00:04.570789 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "0540b11e49f44e6ca238bd5a2c047f8e"
format_stamp: "Formatted at 2026-09-29 03:00:04 on dist-test-slave-q5f0"
server_key: "971625a8082c895514809073b2667f61"
server_key_iv: "f21b14204b451e717bf937fada589b43"
server_key_version: "encryptionkey@0"
I20260929 03:00:04.576308 2353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 03:00:04.579990 2368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:04.581569 2353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 03:00:04.581737 2353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "0540b11e49f44e6ca238bd5a2c047f8e"
format_stamp: "Formatted at 2026-09-29 03:00:04 on dist-test-slave-q5f0"
server_key: "971625a8082c895514809073b2667f61"
server_key_iv: "f21b14204b451e717bf937fada589b43"
server_key_version: "encryptionkey@0"
I20260929 03:00:04.581976 2353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:04.606369 2353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:04.610929 2353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:04.611208 2353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:04.612267 2353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:00:04.613660 2353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:00:04.613734 2353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:04.613850 2353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:00:04.613907 2353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:04.645537 2353 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:36285
I20260929 03:00:04.645592 2481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:36285 every 8 connection(s)
I20260929 03:00:04.646883 2353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 03:00:04.648649 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 2353
I20260929 03:00:04.648816 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 03:00:04.649350 15433 external_mini_cluster.cc:1428] Setting key bd3c0f822206a37f3eaaba59984c554b
Sep 29 03:00:04 dist-test-slave-q5f0 krb5kdc[2247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650804, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.65@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 03:00:04.669307 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:04.649627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:42595 (local address 127.15.18.126:45473)
0929 03:00:04.650076 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:04.650081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:04.651313 (+ 1232us) server_negotiation.cc:407] Connection header received
0929 03:00:04.653809 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:04.653813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:04.653881 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:04.653975 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:04.656026 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:04.656786 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:04.658937 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:04.659378 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:04.659482 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:04.663292 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:04.663298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:04.663301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:04.663336 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:04.665822 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:04.666777 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:04.666781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:04.666784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:04.666914 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:04.667453 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:04.667457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:04.667460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:04.667835 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:04.667966 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:04.668845 (+ 879us) server_negotiation.cc:299] Negotiation successful
0929 03:00:04.669031 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":245,"threads_started":1}
I20260929 03:00:04.670653 2485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:04.650278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45473 (local address 127.15.18.65:42595)
0929 03:00:04.651089 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:04.651160 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:04.653484 (+ 2324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:04.654248 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:04.654259 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:04.655008 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:04.655817 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:04.655842 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:04.656956 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:04.656961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:04.658764 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:04.658779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:04.659642 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:04.659650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:04.659919 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:04.660649 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:04.660680 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:04.663045 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:04.665979 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:04.665989 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:04.666009 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:04.666586 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:04.667028 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:04.667032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:04.667036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:04.667284 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:04.667971 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:04.667979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:04.668406 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 03:00:04.668693 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 03:00:04.669238 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":344,"threads_started":1}
I20260929 03:00:04.672429 2482 heartbeater.cc:347] Connected to a master server at 127.15.18.126:45473
I20260929 03:00:04.672798 2482 heartbeater.cc:464] Registering TS with master...
I20260929 03:00:04.673727 2482 heartbeater.cc:511] Master 127.15.18.126:45473 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.18.66@KRBTEST.COM; defaulting to no policy
I20260929 03:00:04.676200 2292 ts_manager.cc:194] Registered new tserver with Master: 0540b11e49f44e6ca238bd5a2c047f8e (127.15.18.65:36285)
I20260929 03:00:04.677724 2292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.65@KRBTEST.COM'} at 127.15.18.65:42595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
Entry for principal kudu/127.15.18.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab.
I20260929 03:00:04.707834 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45473
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:05.033638 2488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:05.033946 2488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:05.034096 2488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:05.041509 2488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:05.041646 2488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:05.041805 2488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 03:00:05.051569 2488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.66.keytab
--principal=kudu/127.15.18.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:05.052888 2488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:05.054209 2488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:05.068642 2494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:05.068729 2493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:05.068935 2496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:05.069787 2488 server_base.cc:1034] running on GCE node
I20260929 03:00:05.070453 2488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:05.071448 2488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:05.072669 2488 hybrid_clock.cc:648] HybridClock initialized: now 1790650805072637 us; error 78 us; skew 500 ppm
I20260929 03:00:05.072985 2488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:05 dist-test-slave-q5f0 krb5kdc[2247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650805, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:05.076164 2488 init.cc:377] Logged in from keytab as kudu/127.15.18.66@KRBTEST.COM (short username kudu)
I20260929 03:00:05.077991 2488 webserver.cc:460] Webserver started at http://127.15.18.66:42929/ using document root <none> and password file <none>
I20260929 03:00:05.078720 2488 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:05.078852 2488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:05.079239 2488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:05.082549 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "5d586189604d4d9ea6301bd171ba4e9b"
format_stamp: "Formatted at 2026-09-29 03:00:05 on dist-test-slave-q5f0"
server_key: "2518ca24c925611730f1c3d5e5b04345"
server_key_iv: "09fae43e9e2d35c0b41f9a4154a1b6ea"
server_key_version: "encryptionkey@0"
I20260929 03:00:05.083410 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "5d586189604d4d9ea6301bd171ba4e9b"
format_stamp: "Formatted at 2026-09-29 03:00:05 on dist-test-slave-q5f0"
server_key: "2518ca24c925611730f1c3d5e5b04345"
server_key_iv: "09fae43e9e2d35c0b41f9a4154a1b6ea"
server_key_version: "encryptionkey@0"
I20260929 03:00:05.089258 2488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:00:05.093160 2503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:05.094640 2488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260929 03:00:05.094797 2488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "5d586189604d4d9ea6301bd171ba4e9b"
format_stamp: "Formatted at 2026-09-29 03:00:05 on dist-test-slave-q5f0"
server_key: "2518ca24c925611730f1c3d5e5b04345"
server_key_iv: "09fae43e9e2d35c0b41f9a4154a1b6ea"
server_key_version: "encryptionkey@0"
I20260929 03:00:05.095036 2488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:05.117328 2488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:05.121593 2488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:05.121870 2488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:05.122843 2488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:00:05.124293 2488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:00:05.124368 2488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:05.124486 2488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:00:05.124533 2488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:05.156137 2488 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:42825
I20260929 03:00:05.156205 2616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:42825 every 8 connection(s)
I20260929 03:00:05.157383 2488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 03:00:05.162559 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 2488
I20260929 03:00:05.162751 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 03:00:05.163244 15433 external_mini_cluster.cc:1428] Setting key 0f32e00ee30f4b3d1adbe9ffcf9a696f
Sep 29 03:00:05 dist-test-slave-q5f0 krb5kdc[2247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650805, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.66@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 03:00:05.177918 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.159831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:41381 (local address 127.15.18.126:45473)
0929 03:00:05.160060 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:05.160067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:05.161213 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 03:00:05.163056 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:05.163062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:05.163154 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:05.163267 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:05.164871 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.165570 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.167154 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.167645 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.167759 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.171470 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:05.171476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:05.171481 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:05.171553 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:05.174562 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:05.175496 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:05.175500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:05.175502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:05.175681 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:05.176189 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:05.176192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:05.176194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:05.176475 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:05.176579 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:05.177465 (+ 886us) server_negotiation.cc:299] Negotiation successful
0929 03:00:05.177667 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 03:00:05.179268 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.160326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45473 (local address 127.15.18.66:41381)
0929 03:00:05.161041 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:05.161079 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:05.162732 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:05.163551 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:05.163560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:05.164087 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:05.164663 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:05.164679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.165740 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.165744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:05.166989 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:05.167001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.167872 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.167881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:05.168108 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.168790 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:05.168815 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:05.171170 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:05.174750 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:05.174760 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:05.174779 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:05.175384 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:05.175790 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:05.175795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:05.175798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:05.176069 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:05.176611 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:05.176618 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:05.177059 (+ 441us) client_negotiation.cc:769] Sending connection context
0929 03:00:05.177348 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 03:00:05.177911 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":270,"threads_started":1}
I20260929 03:00:05.180835 2617 heartbeater.cc:347] Connected to a master server at 127.15.18.126:45473
I20260929 03:00:05.181146 2617 heartbeater.cc:464] Registering TS with master...
I20260929 03:00:05.181874 2617 heartbeater.cc:511] Master 127.15.18.126:45473 requested a full tablet report, sending...
I20260929 03:00:05.183394 2292 ts_manager.cc:194] Registered new tserver with Master: 5d586189604d4d9ea6301bd171ba4e9b (127.15.18.66:42825)
I20260929 03:00:05.184293 2292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.66@KRBTEST.COM'} at 127.15.18.66:41381
WARNING: no policy specified for kudu/127.15.18.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
Entry for principal kudu/127.15.18.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab.
I20260929 03:00:05.220559 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45473
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:05.543704 2622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:05.543991 2622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:05.544122 2622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:05.551474 2622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:05.551635 2622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:05.551803 2622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 03:00:05.561776 2622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.67.keytab
--principal=kudu/127.15.18.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:45473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:05.563074 2622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:05.564412 2622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:05.577497 2627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:05.577593 2628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:05.578150 2630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:05.578281 2622 server_base.cc:1034] running on GCE node
I20260929 03:00:05.578907 2622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:05.580204 2622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:05.581468 2622 hybrid_clock.cc:648] HybridClock initialized: now 1790650805581397 us; error 51 us; skew 500 ppm
I20260929 03:00:05.581784 2622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:05 dist-test-slave-q5f0 krb5kdc[2247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650805, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:05.585057 2622 init.cc:377] Logged in from keytab as kudu/127.15.18.67@KRBTEST.COM (short username kudu)
I20260929 03:00:05.586675 2622 webserver.cc:460] Webserver started at http://127.15.18.67:36201/ using document root <none> and password file <none>
I20260929 03:00:05.587428 2622 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:05.587579 2622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:05.587874 2622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:05.590826 2622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "088b45fb5f1547ec9c0e8820493ab798"
format_stamp: "Formatted at 2026-09-29 03:00:05 on dist-test-slave-q5f0"
server_key: "aebf3caf7d5896a12c896dc3130fb6bd"
server_key_iv: "86e9af229ec3501a47efe749d069757c"
server_key_version: "encryptionkey@0"
I20260929 03:00:05.591614 2622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "088b45fb5f1547ec9c0e8820493ab798"
format_stamp: "Formatted at 2026-09-29 03:00:05 on dist-test-slave-q5f0"
server_key: "aebf3caf7d5896a12c896dc3130fb6bd"
server_key_iv: "86e9af229ec3501a47efe749d069757c"
server_key_version: "encryptionkey@0"
I20260929 03:00:05.596868 2622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 03:00:05.600622 2637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:05.602061 2622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:00:05.602222 2622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "088b45fb5f1547ec9c0e8820493ab798"
format_stamp: "Formatted at 2026-09-29 03:00:05 on dist-test-slave-q5f0"
server_key: "aebf3caf7d5896a12c896dc3130fb6bd"
server_key_iv: "86e9af229ec3501a47efe749d069757c"
server_key_version: "encryptionkey@0"
I20260929 03:00:05.602409 2622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:05.617647 2622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:05.621982 2622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:05.622291 2622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:05.623354 2622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:00:05.624706 2622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:00:05.624785 2622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:05.624876 2622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:00:05.624930 2622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:05.656695 2622 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:33451
I20260929 03:00:05.656739 2750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:33451 every 8 connection(s)
I20260929 03:00:05.657919 2622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 03:00:05.664966 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 2622
I20260929 03:00:05.665146 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 03:00:05.665702 15433 external_mini_cluster.cc:1428] Setting key 849516855772bc8b06a347e939259c97
Sep 29 03:00:05 dist-test-slave-q5f0 krb5kdc[2247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650805, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.67@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 03:00:05.678120 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.660363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:44623 (local address 127.15.18.126:45473)
0929 03:00:05.660587 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:05.660593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:05.661733 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 03:00:05.663468 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:05.663473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:05.663609 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:05.663701 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:05.665241 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.666277 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.668145 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.668678 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.668797 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.672348 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:05.672353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:05.672357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:05.672391 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:05.674955 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:05.675774 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:05.675779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:05.675782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:05.675914 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:05.676369 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:05.676373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:05.676380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:05.676666 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:05.676808 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:05.677542 (+ 734us) server_negotiation.cc:299] Negotiation successful
0929 03:00:05.677784 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:00:05.679083 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.660847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:45473 (local address 127.15.18.67:44623)
0929 03:00:05.661549 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:05.661598 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:05.663202 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:05.663924 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:05.663932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:05.664447 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:05.665013 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:05.665030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.666458 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.666463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:05.667991 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:05.668005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.668915 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.668922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:05.669143 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.669811 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:05.669834 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:05.672110 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:05.675132 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:05.675139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:05.675152 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:05.675642 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:05.676032 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:05.676037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:05.676041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:05.676228 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:05.676797 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:05.676805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:05.677156 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 03:00:05.677402 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 03:00:05.677952 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":276,"threads_started":1}
I20260929 03:00:05.680620 2751 heartbeater.cc:347] Connected to a master server at 127.15.18.126:45473
I20260929 03:00:05.680956 2751 heartbeater.cc:464] Registering TS with master...
I20260929 03:00:05.681087 2482 heartbeater.cc:503] Master 127.15.18.126:45473 was elected leader, sending a full tablet report...
I20260929 03:00:05.681883 2751 heartbeater.cc:511] Master 127.15.18.126:45473 requested a full tablet report, sending...
I20260929 03:00:05.683312 2291 ts_manager.cc:194] Registered new tserver with Master: 088b45fb5f1547ec9c0e8820493ab798 (127.15.18.67:33451)
I20260929 03:00:05.684186 2291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.18.67@KRBTEST.COM'} at 127.15.18.67:44623
I20260929 03:00:05.694181 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:00:05.712052 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.698404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.15.18.126:45473)
0929 03:00:05.698620 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:05.698627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:05.699057 (+ 430us) server_negotiation.cc:407] Connection header received
0929 03:00:05.699197 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:05.699201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:05.699266 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:05.699346 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:05.700938 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.701718 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.703126 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.703603 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.703724 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.706199 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:05.706205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:05.706208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:05.706259 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:05.709080 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:05.709898 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:05.709902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:05.709904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:05.710042 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:05.710468 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:05.710472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:05.710475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:05.710887 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:05.711090 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:05.711461 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 03:00:05.711705 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:00:05.716796 2291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:00:05.723906 2291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:00:05.748639 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.737133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:33451 (local address 127.0.0.1:34840)
0929 03:00:05.738460 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:05.738506 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:05.738687 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:05.741047 (+ 2360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:05.741054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:05.741257 (+ 203us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:05.741633 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:05.741646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.743360 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.743365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:05.745651 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:05.745665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.747803 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.747809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:05.747984 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.748018 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 03:00:05.748098 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:00:05.748246 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":1871,"threads_started":1}
I20260929 03:00:05.748868 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.737794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.15.18.65:36285)
0929 03:00:05.741206 (+ 3412us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:05.741214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:05.741242 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:00:05.741331 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:05.741336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:05.741497 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:05.741667 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:05.742312 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.743960 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.746302 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.747262 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.747384 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.747649 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:05.748368 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 03:00:05.748530 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2028,"thread_start_us":1815,"threads_started":1}
I20260929 03:00:05.748914 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.740048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:42825 (local address 127.0.0.1:59492)
0929 03:00:05.740623 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:05.740639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:05.740740 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:05.741735 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:05.741740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:05.741766 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:05.742220 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:05.742232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.744142 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.744146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:05.746271 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:05.746285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.748048 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.748054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:05.748211 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.748242 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:00:05.748296 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:00:05.748413 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":25,"thread_start_us":652,"threads_started":1}
I20260929 03:00:05.748996 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.737169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:36285 (local address 127.0.0.1:34442)
0929 03:00:05.739258 (+ 2089us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:05.739275 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:05.739373 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:05.741681 (+ 2308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:05.741686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:05.741709 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:05.742116 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:05.742125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.744122 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.744127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:05.746120 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:05.746134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.747463 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.747469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:05.747674 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.747726 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 03:00:05.747886 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 03:00:05.748096 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1960,"mutex_wait_us":78,"thread_start_us":856,"threads_started":1}
I20260929 03:00:05.749936 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.740554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.15.18.66:42825)
0929 03:00:05.741167 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:05.741175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:05.741201 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:00:05.741304 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:05.741312 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:05.741530 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:05.742340 (+ 810us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:05.742539 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.744009 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.746427 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.747833 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.749039 (+ 1206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.749533 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:05.749599 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:00:05.749725 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":354,"threads_started":1}
I20260929 03:00:05.750571 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:05.737268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.15.18.67:33451)
0929 03:00:05.740372 (+ 3104us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:05.740382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:05.740417 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:00:05.740529 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:05.740537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:05.740813 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:05.741843 (+ 1030us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:05.742097 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.743212 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.745832 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:05.747279 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:05.747400 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:05.747651 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:05.749208 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0929 03:00:05.750236 (+ 1028us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2951,"thread_start_us":2812,"threads_started":1}
I20260929 03:00:05.752661 2417 tablet_service.cc:1467] Processing CreateTablet for tablet a7f05d170c224118b42c4cd21a2cf4bb (DEFAULT_TABLE table=test-table [id=c8dfe85f59444d0e96f613b053ac296e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:00:05.753366 2552 tablet_service.cc:1467] Processing CreateTablet for tablet a7f05d170c224118b42c4cd21a2cf4bb (DEFAULT_TABLE table=test-table [id=c8dfe85f59444d0e96f613b053ac296e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:00:05.754184 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f05d170c224118b42c4cd21a2cf4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:05.754045 2686 tablet_service.cc:1467] Processing CreateTablet for tablet a7f05d170c224118b42c4cd21a2cf4bb (DEFAULT_TABLE table=test-table [id=c8dfe85f59444d0e96f613b053ac296e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:00:05.754724 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f05d170c224118b42c4cd21a2cf4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:05.755357 2686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f05d170c224118b42c4cd21a2cf4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:05.765151 2769 tablet_bootstrap.cc:492] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: Bootstrap starting.
I20260929 03:00:05.769454 2769 tablet_bootstrap.cc:654] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:05.769906 2771 tablet_bootstrap.cc:492] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: Bootstrap starting.
I20260929 03:00:05.770731 2770 tablet_bootstrap.cc:492] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: Bootstrap starting.
I20260929 03:00:05.770907 2769 log.cc:824] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:05.773972 2770 tablet_bootstrap.cc:654] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:05.774057 2771 tablet_bootstrap.cc:654] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:05.775017 2769 tablet_bootstrap.cc:492] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: No bootstrap required, opened a new log
I20260929 03:00:05.774974 2770 log.cc:824] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:05.775300 2771 log.cc:824] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:05.775347 2769 ts_tablet_manager.cc:1397] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260929 03:00:05.779032 2771 tablet_bootstrap.cc:492] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: No bootstrap required, opened a new log
I20260929 03:00:05.779246 2770 tablet_bootstrap.cc:492] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: No bootstrap required, opened a new log
I20260929 03:00:05.779305 2771 ts_tablet_manager.cc:1397] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 03:00:05.779498 2770 ts_tablet_manager.cc:1397] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260929 03:00:05.785750 2769 raft_consensus.cc:348] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b45fb5f1547ec9c0e8820493ab798" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33451 } } peers { permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 } } peers { permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 } }
I20260929 03:00:05.786284 2769 raft_consensus.cc:374] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:05.786434 2769 raft_consensus.cc:729] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0540b11e49f44e6ca238bd5a2c047f8e, State: Initialized, Role: FOLLOWER
I20260929 03:00:05.786841 2771 raft_consensus.cc:348] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b45fb5f1547ec9c0e8820493ab798" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33451 } } peers { permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 } } peers { permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 } }
I20260929 03:00:05.787122 2769 consensus_queue.cc:260] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b45fb5f1547ec9c0e8820493ab798" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33451 } } peers { permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 } } peers { permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 } }
I20260929 03:00:05.787325 2771 raft_consensus.cc:374] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:05.787424 2771 raft_consensus.cc:729] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d586189604d4d9ea6301bd171ba4e9b, State: Initialized, Role: FOLLOWER
I20260929 03:00:05.789280 2769 ts_tablet_manager.cc:1428] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 03:00:05.788237 2771 consensus_queue.cc:260] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b45fb5f1547ec9c0e8820493ab798" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33451 } } peers { permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 } } peers { permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 } }
I20260929 03:00:05.789546 2770 raft_consensus.cc:348] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b45fb5f1547ec9c0e8820493ab798" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33451 } } peers { permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 } } peers { permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 } }
I20260929 03:00:05.790028 2770 raft_consensus.cc:374] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:05.790158 2770 raft_consensus.cc:729] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088b45fb5f1547ec9c0e8820493ab798, State: Initialized, Role: FOLLOWER
I20260929 03:00:05.790470 2617 heartbeater.cc:503] Master 127.15.18.126:45473 was elected leader, sending a full tablet report...
I20260929 03:00:05.790859 2770 consensus_queue.cc:260] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b45fb5f1547ec9c0e8820493ab798" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33451 } } peers { permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 } } peers { permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 } }
I20260929 03:00:05.791098 2771 ts_tablet_manager.cc:1428] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: Time spent starting tablet: real 0.012s user 0.005s sys 0.004s
I20260929 03:00:05.792114 2751 heartbeater.cc:503] Master 127.15.18.126:45473 was elected leader, sending a full tablet report...
I20260929 03:00:05.792673 2770 ts_tablet_manager.cc:1428] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
W20260929 03:00:05.899416 2483 tablet.cc:2369] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:00:05.909554 2618 tablet.cc:2369] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:00:05.909554 2752 tablet.cc:2369] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:00:06.018970 2777 raft_consensus.cc:482] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:00:06.019299 2777 raft_consensus.cc:504] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b45fb5f1547ec9c0e8820493ab798" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33451 } } peers { permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 } } peers { permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 } }
I20260929 03:00:06.021034 2777 leader_election.cc:288] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0540b11e49f44e6ca238bd5a2c047f8e (127.15.18.65:36285), 5d586189604d4d9ea6301bd171ba4e9b (127.15.18.66:42825)
I20260929 03:00:06.027580 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:06.021514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:36285 (local address 127.15.18.67:44687)
0929 03:00:06.021691 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:06.021711 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:06.021830 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:06.022434 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:06.022440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:06.022473 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:06.022707 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:06.022717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.024119 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.024123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:06.025644 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:06.025659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.026929 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.026934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:06.027082 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:06.027108 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 03:00:06.027188 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:00:06.027317 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":37}
I20260929 03:00:06.027643 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:06.021940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:44687 (local address 127.15.18.65:36285)
0929 03:00:06.022128 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:06.022133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:06.022149 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:00:06.022212 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:06.022217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:06.022280 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:06.022358 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:06.022881 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.023959 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.025872 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.026759 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.026931 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:06.027005 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:06.027244 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 03:00:06.027377 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:00:06.028889 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:06.021495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:47409 (local address 127.15.18.66:42825)
0929 03:00:06.021680 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:06.021684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:06.023093 (+ 1409us) server_negotiation.cc:407] Connection header received
0929 03:00:06.023324 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:06.023330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:06.023395 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:06.023470 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:06.024122 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.024908 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.026793 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.027965 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.028118 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:06.028185 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:06.028528 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 03:00:06.028659 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:00:06.028945 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:06.022132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:42825 (local address 127.15.18.67:47409)
0929 03:00:06.022979 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:06.022996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:06.023120 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:06.023540 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:06.023545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:06.023571 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:06.023918 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:06.023930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.025070 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.025076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:06.026607 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:06.026619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.028164 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.028170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:06.028322 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:06.028356 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 03:00:06.028468 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:00:06.028591 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":19,"thread_start_us":469,"threads_started":1}
I20260929 03:00:06.028851 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f05d170c224118b42c4cd21a2cf4bb" candidate_uuid: "088b45fb5f1547ec9c0e8820493ab798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" is_pre_election: true
I20260929 03:00:06.029453 2437 raft_consensus.cc:2454] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088b45fb5f1547ec9c0e8820493ab798 in term 0.
I20260929 03:00:06.030157 2641 leader_election.cc:302] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0540b11e49f44e6ca238bd5a2c047f8e, 088b45fb5f1547ec9c0e8820493ab798; no voters:
I20260929 03:00:06.030125 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f05d170c224118b42c4cd21a2cf4bb" candidate_uuid: "088b45fb5f1547ec9c0e8820493ab798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d586189604d4d9ea6301bd171ba4e9b" is_pre_election: true
I20260929 03:00:06.030525 2777 raft_consensus.cc:2781] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:00:06.030643 2777 raft_consensus.cc:482] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:00:06.030616 2572 raft_consensus.cc:2454] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088b45fb5f1547ec9c0e8820493ab798 in term 0.
I20260929 03:00:06.030733 2777 raft_consensus.cc:3037] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:06.033196 2777 raft_consensus.cc:504] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b45fb5f1547ec9c0e8820493ab798" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33451 } } peers { permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 } } peers { permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 } }
I20260929 03:00:06.033775 2777 leader_election.cc:288] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [CANDIDATE]: Term 1 election: Requested vote from peers 0540b11e49f44e6ca238bd5a2c047f8e (127.15.18.65:36285), 5d586189604d4d9ea6301bd171ba4e9b (127.15.18.66:42825)
I20260929 03:00:06.034302 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f05d170c224118b42c4cd21a2cf4bb" candidate_uuid: "088b45fb5f1547ec9c0e8820493ab798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d586189604d4d9ea6301bd171ba4e9b"
I20260929 03:00:06.034302 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f05d170c224118b42c4cd21a2cf4bb" candidate_uuid: "088b45fb5f1547ec9c0e8820493ab798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0540b11e49f44e6ca238bd5a2c047f8e"
I20260929 03:00:06.034504 2572 raft_consensus.cc:3037] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:06.034504 2437 raft_consensus.cc:3037] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:06.037598 2437 raft_consensus.cc:2454] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088b45fb5f1547ec9c0e8820493ab798 in term 1.
I20260929 03:00:06.037598 2572 raft_consensus.cc:2454] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088b45fb5f1547ec9c0e8820493ab798 in term 1.
I20260929 03:00:06.038013 2638 leader_election.cc:302] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088b45fb5f1547ec9c0e8820493ab798, 5d586189604d4d9ea6301bd171ba4e9b; no voters:
I20260929 03:00:06.038281 2777 raft_consensus.cc:2781] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:06.038718 2777 raft_consensus.cc:686] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 1 LEADER]: Becoming Leader. State: Replica: 088b45fb5f1547ec9c0e8820493ab798, State: Running, Role: LEADER
I20260929 03:00:06.039189 2777 consensus_queue.cc:237] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b45fb5f1547ec9c0e8820493ab798" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33451 } } peers { permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 } } peers { permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 } }
I20260929 03:00:06.044245 2290 catalog_manager.cc:5495] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088b45fb5f1547ec9c0e8820493ab798 (127.15.18.67). New cstate: current_term: 1 leader_uuid: "088b45fb5f1547ec9c0e8820493ab798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088b45fb5f1547ec9c0e8820493ab798" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 33451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 } health_report { overall_health: UNKNOWN } } }
I20260929 03:00:06.115454 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:06.110098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.15.18.67:33451)
0929 03:00:06.110306 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:06.110311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:06.110333 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:00:06.110532 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:06.110536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:06.110615 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:06.110738 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:06.111071 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.111851 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.113225 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.113628 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.113722 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:06.114293 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:06.114823 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:06.114953 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:06.115112 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:00:06.115183 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 03:00:06.123770 2572 raft_consensus.cc:1261] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b [term 1 FOLLOWER]: Refusing update from remote peer 088b45fb5f1547ec9c0e8820493ab798: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:00:06.123769 2437 raft_consensus.cc:1261] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e [term 1 FOLLOWER]: Refusing update from remote peer 088b45fb5f1547ec9c0e8820493ab798: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:00:06.124795 2777 consensus_queue.cc:1035] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d586189604d4d9ea6301bd171ba4e9b" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 42825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:00:06.125093 2779 consensus_queue.cc:1035] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0540b11e49f44e6ca238bd5a2c047f8e" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 36285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:00:06.138875 2784 mvcc.cc:205] Tried to move back new op lower bound from 7334505701871861760 to 7334505701540528128. Current Snapshot: MvccSnapshot[applied={T|T < 7334505701871861760}]
I20260929 03:00:06.149997 2291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:00:06.150262 2291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:00:06.153829 2291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9272169e3da04c0487d6848ad587d496: Sending DeleteTablet for 3 replicas of tablet a7f05d170c224118b42c4cd21a2cf4bb
I20260929 03:00:06.155042 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f05d170c224118b42c4cd21a2cf4bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:00:06 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:59492
I20260929 03:00:06.155042 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f05d170c224118b42c4cd21a2cf4bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:00:06 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:34442
I20260929 03:00:06.155074 2686 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f05d170c224118b42c4cd21a2cf4bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:00:06 UTC) from {username='kudu', principal='kudu/127.15.18.126@KRBTEST.COM'} at 127.0.0.1:34840
I20260929 03:00:06.156320 2793 tablet_replica.cc:330] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: stopping tablet replica
I20260929 03:00:06.156790 2793 raft_consensus.cc:2229] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:00:06.157326 2793 raft_consensus.cc:2258] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:00:06.158138 2792 tablet_replica.cc:330] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: stopping tablet replica
I20260929 03:00:06.158560 2792 raft_consensus.cc:2229] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:00:06.159343 2792 raft_consensus.cc:2258] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:00:06.159569 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 2353
I20260929 03:00:06.161000 2794 tablet_replica.cc:330] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: stopping tablet replica
I20260929 03:00:06.161408 2794 raft_consensus.cc:2229] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:00:06.162056 2794 raft_consensus.cc:2258] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:00:06.163734 2793 ts_tablet_manager.cc:1905] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:00:06.166146 2794 ts_tablet_manager.cc:1905] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:00:06.169214 2792 ts_tablet_manager.cc:1905] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:00:06.170918 2793 ts_tablet_manager.cc:1918] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:00:06.171128 2793 log.cc:1196] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/a7f05d170c224118b42c4cd21a2cf4bb
I20260929 03:00:06.171725 2793 ts_tablet_manager.cc:1939] T a7f05d170c224118b42c4cd21a2cf4bb P 5d586189604d4d9ea6301bd171ba4e9b: Deleting consensus metadata
I20260929 03:00:06.174168 2276 catalog_manager.cc:4841] TS 5d586189604d4d9ea6301bd171ba4e9b (127.15.18.66:42825): tablet a7f05d170c224118b42c4cd21a2cf4bb (table test-table [id=c8dfe85f59444d0e96f613b053ac296e]) successfully deleted
I20260929 03:00:06.176059 2792 ts_tablet_manager.cc:1918] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:00:06.176221 2792 log.cc:1196] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/a7f05d170c224118b42c4cd21a2cf4bb
I20260929 03:00:06.176685 2792 ts_tablet_manager.cc:1939] T a7f05d170c224118b42c4cd21a2cf4bb P 0540b11e49f44e6ca238bd5a2c047f8e: Deleting consensus metadata
I20260929 03:00:06.178450 2279 catalog_manager.cc:4841] TS 0540b11e49f44e6ca238bd5a2c047f8e (127.15.18.65:36285): tablet a7f05d170c224118b42c4cd21a2cf4bb (table test-table [id=c8dfe85f59444d0e96f613b053ac296e]) successfully deleted
Sep 29 03:00:06 dist-test-slave-q5f0 krb5kdc[2247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.65@KRBTEST.COM
I20260929 03:00:06.184358 2794 ts_tablet_manager.cc:1918] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:00:06.184566 2794 log.cc:1196] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/a7f05d170c224118b42c4cd21a2cf4bb
I20260929 03:00:06.185132 2794 ts_tablet_manager.cc:1939] T a7f05d170c224118b42c4cd21a2cf4bb P 088b45fb5f1547ec9c0e8820493ab798: Deleting consensus metadata
I20260929 03:00:06.187063 2276 catalog_manager.cc:4841] TS 088b45fb5f1547ec9c0e8820493ab798 (127.15.18.67:33451): tablet a7f05d170c224118b42c4cd21a2cf4bb (table test-table [id=c8dfe85f59444d0e96f613b053ac296e]) successfully deleted
I20260929 03:00:06.188444 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:06.160853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.15.18.65:36285)
0929 03:00:06.161019 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:06.161024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:06.161825 (+ 801us) server_negotiation.cc:407] Connection header received
0929 03:00:06.162131 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:06.162136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:06.162219 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:06.162305 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:06.170799 (+ 8494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.172037 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.174052 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.174706 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.174821 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:06.180374 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:06.180389 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:06.180407 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:06.180462 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:06.184346 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:06.185532 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:06.185543 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:06.185547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:06.185764 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:06.186277 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:06.186282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:06.186286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:06.186778 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:06.187099 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:06.187817 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 03:00:06.188063 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:00:06.189157 2477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:06.281780 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 2353
I20260929 03:00:06.303400 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 2488
Sep 29 03:00:06 dist-test-slave-q5f0 krb5kdc[2247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.66@KRBTEST.COM
I20260929 03:00:06.320760 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:06.304249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.15.18.66:42825)
0929 03:00:06.304457 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:06.304462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:06.304481 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:00:06.304614 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:06.304619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:06.304688 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:06.304790 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:06.306097 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.306886 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.308482 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.308897 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.309006 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:06.313664 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:06.313686 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:06.313709 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:06.313759 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:06.317541 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:06.318418 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:06.318428 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:06.318432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:06.318600 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:06.319043 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:06.319048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:06.319052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:06.319561 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:06.319896 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:06.320153 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 03:00:06.320385 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 03:00:06.321532 2612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:06.413600 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 2488
I20260929 03:00:06.434206 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 2622
Sep 29 03:00:06 dist-test-slave-q5f0 krb5kdc[2247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.67@KRBTEST.COM
I20260929 03:00:06.451623 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:06.435080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.15.18.67:33451)
0929 03:00:06.435284 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:06.435291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:06.435329 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:00:06.435400 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:06.435409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:06.435550 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:06.435662 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:06.437083 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.438389 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.439927 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:06.440399 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:06.440525 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:06.445215 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:06.445227 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:06.445243 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:06.445290 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:06.448338 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:06.449062 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:06.449069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:06.449072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:06.449227 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:06.449649 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:06.449653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:06.449655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:06.450086 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:06.450402 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:06.450992 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 03:00:06.451218 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 03:00:06.452320 2746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:06.574832 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 2622
I20260929 03:00:06.590653 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 2260
I20260929 03:00:06.591801 2322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:06.703176 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 2260
2026-09-29T03:00:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4441 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T03:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:06Z Disabled control of system clock
I20260929 03:00:06.774827 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:39443
--webserver_interface=127.15.18.126
--webserver_port=0
--builtin_ntp_servers=127.15.18.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:39443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:00:07.106565 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:07.106902 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:07.107008 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:07.114544 2804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:07.114681 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:07.114732 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:07.114814 2804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:07.114867 2804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:07.124549 2804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:39443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:39443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:07.125895 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:07.127709 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:07.139235 2809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:07.139197 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:07.139271 2810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:07.139935 2804 server_base.cc:1034] running on GCE node
I20260929 03:00:07.140810 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:07.142419 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:07.143649 2804 hybrid_clock.cc:648] HybridClock initialized: now 1790650807143570 us; error 69 us; skew 500 ppm
I20260929 03:00:07.144145 2804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:00:07.153605 2804 webserver.cc:460] Webserver started at http://127.15.18.126:46695/ using document root <none> and password file <none>
I20260929 03:00:07.154402 2804 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:07.154518 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:07.154851 2804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:07.158102 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "941dffae6bba4cc3812e513cfa78adbf"
format_stamp: "Formatted at 2026-09-29 03:00:07 on dist-test-slave-q5f0"
server_key: "004575b628e7040f8c3d1969c4da9ff1"
server_key_iv: "2c16f50d7efaf7ebd44f67eb2f348027"
server_key_version: "encryptionkey@0"
I20260929 03:00:07.159039 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "941dffae6bba4cc3812e513cfa78adbf"
format_stamp: "Formatted at 2026-09-29 03:00:07 on dist-test-slave-q5f0"
server_key: "004575b628e7040f8c3d1969c4da9ff1"
server_key_iv: "2c16f50d7efaf7ebd44f67eb2f348027"
server_key_version: "encryptionkey@0"
I20260929 03:00:07.164973 2804 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260929 03:00:07.169108 2818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:07.170750 2804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:00:07.170955 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "941dffae6bba4cc3812e513cfa78adbf"
format_stamp: "Formatted at 2026-09-29 03:00:07 on dist-test-slave-q5f0"
server_key: "004575b628e7040f8c3d1969c4da9ff1"
server_key_iv: "2c16f50d7efaf7ebd44f67eb2f348027"
server_key_version: "encryptionkey@0"
I20260929 03:00:07.171187 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:07.199637 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:07.200541 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:07.200824 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:07.219597 2804 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:39443
I20260929 03:00:07.219759 2869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:39443 every 8 connection(s)
I20260929 03:00:07.220999 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:07.224860 2870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:07.230455 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 2804
I20260929 03:00:07.230615 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:07.231056 15433 external_mini_cluster.cc:1428] Setting key 2a6f5f9c02cd2e25a6173343eef0b5db
I20260929 03:00:07.236236 2870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf: Bootstrap starting.
I20260929 03:00:07.239499 2870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:07.240584 2870 log.cc:824] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:07.243831 2870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf: No bootstrap required, opened a new log
I20260929 03:00:07.246366 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:07.233037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.15.18.126:39443)
0929 03:00:07.234200 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:07.234212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:07.234265 (+ 53us) server_negotiation.cc:407] Connection header received
0929 03:00:07.236204 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:07.236235 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:07.236801 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:07.237222 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:07.237945 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:07.239332 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:07.240994 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:07.242177 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:07.242373 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:07.243665 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:07.243676 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:07.243693 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:07.244422 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:07.244523 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:07.244550 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:07.244670 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:07.244924 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 03:00:07.245265 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":399,"threads_started":1}
I20260929 03:00:07.252338 2870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941dffae6bba4cc3812e513cfa78adbf" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39443 } }
I20260929 03:00:07.252727 2870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:07.252847 2870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941dffae6bba4cc3812e513cfa78adbf, State: Initialized, Role: FOLLOWER
I20260929 03:00:07.253443 2870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941dffae6bba4cc3812e513cfa78adbf" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39443 } }
I20260929 03:00:07.253679 2870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:07.253824 2870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:07.253997 2870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:07.256479 2870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941dffae6bba4cc3812e513cfa78adbf" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39443 } }
I20260929 03:00:07.257009 2870 leader_election.cc:302] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 941dffae6bba4cc3812e513cfa78adbf; no voters:
I20260929 03:00:07.257598 2870 leader_election.cc:288] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:07.257745 2875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:07.258435 2875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [term 1 LEADER]: Becoming Leader. State: Replica: 941dffae6bba4cc3812e513cfa78adbf, State: Running, Role: LEADER
I20260929 03:00:07.258985 2875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941dffae6bba4cc3812e513cfa78adbf" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39443 } }
I20260929 03:00:07.259680 2870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:07.261652 2876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "941dffae6bba4cc3812e513cfa78adbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941dffae6bba4cc3812e513cfa78adbf" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39443 } } }
I20260929 03:00:07.262049 2876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:07.262879 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 941dffae6bba4cc3812e513cfa78adbf. Latest consensus state: current_term: 1 leader_uuid: "941dffae6bba4cc3812e513cfa78adbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941dffae6bba4cc3812e513cfa78adbf" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39443 } } }
I20260929 03:00:07.263197 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:07.266566 2881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:07.271952 2881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:07.281100 2881 catalog_manager.cc:1347] Generated new cluster ID: 8e2b223b726a4f56b6ed0177d5bb6b18
I20260929 03:00:07.281235 2881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:07.289479 2881 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:07.290807 2881 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:07.297506 2881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf: Generated new TSK 0
I20260929 03:00:07.298514 2881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:07.314072 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.65:0
--local_ip_for_outbound_sockets=127.15.18.65
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39443
--builtin_ntp_servers=127.15.18.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:00:07.703287 2895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:07.703634 2895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:07.703742 2895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:07.711246 2895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:07.711397 2895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:07.711556 2895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.65
I20260929 03:00:07.720921 2895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.18.65
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.65
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:07.722291 2895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:07.724124 2895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:07.737315 2901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:07.737372 2903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:07.737409 2900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:07.738145 2895 server_base.cc:1034] running on GCE node
I20260929 03:00:07.738778 2895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:07.739820 2895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:07.741071 2895 hybrid_clock.cc:648] HybridClock initialized: now 1790650807741012 us; error 46 us; skew 500 ppm
I20260929 03:00:07.741438 2895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:00:07.743896 2895 webserver.cc:460] Webserver started at http://127.15.18.65:38663/ using document root <none> and password file <none>
I20260929 03:00:07.744612 2895 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:07.744714 2895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:07.745031 2895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:07.748059 2895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/data/instance:
uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74"
format_stamp: "Formatted at 2026-09-29 03:00:07 on dist-test-slave-q5f0"
server_key: "676ffa44cafdd3c25e4adf8808e6fdf3"
server_key_iv: "247f1ef74ec83dceef96f38f7e9f9871"
server_key_version: "encryptionkey@0"
I20260929 03:00:07.748799 2895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance:
uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74"
format_stamp: "Formatted at 2026-09-29 03:00:07 on dist-test-slave-q5f0"
server_key: "676ffa44cafdd3c25e4adf8808e6fdf3"
server_key_iv: "247f1ef74ec83dceef96f38f7e9f9871"
server_key_version: "encryptionkey@0"
I20260929 03:00:07.754166 2895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 03:00:07.758105 2909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:07.759671 2895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:00:07.759848 2895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/wal
uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74"
format_stamp: "Formatted at 2026-09-29 03:00:07 on dist-test-slave-q5f0"
server_key: "676ffa44cafdd3c25e4adf8808e6fdf3"
server_key_iv: "247f1ef74ec83dceef96f38f7e9f9871"
server_key_version: "encryptionkey@0"
I20260929 03:00:07.760097 2895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:07.792747 2895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:07.793638 2895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:07.793938 2895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:07.794988 2895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:00:07.796577 2895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:00:07.796677 2895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:07.796788 2895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:00:07.796859 2895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:07.829017 2895 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.65:40425
I20260929 03:00:07.829087 3022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.65:40425 every 8 connection(s)
I20260929 03:00:07.830309 2895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/data/info.pb
I20260929 03:00:07.833936 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 2895
I20260929 03:00:07.834111 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/wal/instance
I20260929 03:00:07.834657 15433 external_mini_cluster.cc:1428] Setting key 4d45d06ee0d7f9e87460f5a222ccd7d9
I20260929 03:00:07.837844 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.66:0
--local_ip_for_outbound_sockets=127.15.18.66
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39443
--builtin_ntp_servers=127.15.18.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:00:07.845069 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:07.832896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.65:56561 (local address 127.15.18.126:39443)
0929 03:00:07.833392 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:07.833397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:07.834614 (+ 1217us) server_negotiation.cc:407] Connection header received
0929 03:00:07.836943 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:07.836949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:07.837036 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:07.837154 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:07.838451 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:07.839487 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:07.841703 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:07.842250 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:07.842404 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:07.844097 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:07.844104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:07.844108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:07.844136 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:07.844188 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:07.844204 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:07.844293 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:07.844612 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 03:00:07.844721 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":265,"threads_started":1}
I20260929 03:00:07.846434 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:07.833336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:39443 (local address 127.15.18.65:56561)
0929 03:00:07.834436 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:07.834477 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:07.836592 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:07.837413 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:07.837423 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:07.837522 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:07.838260 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:07.838281 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:07.839700 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:07.839706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:07.841471 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:07.841495 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:07.842495 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:07.842501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:07.842789 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:07.843632 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:00:07.843635 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:07.843699 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:00:07.843858 (+ 159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:07.844358 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:07.844372 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:07.844381 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:00:07.844584 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 03:00:07.845007 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":379,"threads_started":1}
I20260929 03:00:07.848232 3023 heartbeater.cc:347] Connected to a master server at 127.15.18.126:39443
I20260929 03:00:07.848538 3023 heartbeater.cc:464] Registering TS with master...
I20260929 03:00:07.849308 3023 heartbeater.cc:511] Master 127.15.18.126:39443 requested a full tablet report, sending...
I20260929 03:00:07.851613 2835 ts_manager.cc:194] Registered new tserver with Master: b7f6097d4ffe482cbc461f1bb0d5ee74 (127.15.18.65:40425)
I20260929 03:00:07.853585 2835 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.65:56561
W20260929 03:00:08.165199 3027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:08.165514 3027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:08.165608 3027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:08.173138 3027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:08.173267 3027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:08.173386 3027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.66
I20260929 03:00:08.182938 3027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.18.66
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.66
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:08.184332 3027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:08.186090 3027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:08.199326 3032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:08.199324 3033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:08.200009 3035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:08.200228 3027 server_base.cc:1034] running on GCE node
I20260929 03:00:08.200832 3027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:08.201834 3027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:08.203068 3027 hybrid_clock.cc:648] HybridClock initialized: now 1790650808203009 us; error 54 us; skew 500 ppm
I20260929 03:00:08.203401 3027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:00:08.205803 3027 webserver.cc:460] Webserver started at http://127.15.18.66:37285/ using document root <none> and password file <none>
I20260929 03:00:08.206514 3027 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:08.206607 3027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:08.206915 3027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:08.210081 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/data/instance:
uuid: "155dcc5296f545d09b5b4ca12d35d6ba"
format_stamp: "Formatted at 2026-09-29 03:00:08 on dist-test-slave-q5f0"
server_key: "cfc5262e1cc1f896829c2b90d7d7b09a"
server_key_iv: "b49e6ab41ce75eb993fbb83f125632d4"
server_key_version: "encryptionkey@0"
I20260929 03:00:08.210945 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance:
uuid: "155dcc5296f545d09b5b4ca12d35d6ba"
format_stamp: "Formatted at 2026-09-29 03:00:08 on dist-test-slave-q5f0"
server_key: "cfc5262e1cc1f896829c2b90d7d7b09a"
server_key_iv: "b49e6ab41ce75eb993fbb83f125632d4"
server_key_version: "encryptionkey@0"
I20260929 03:00:08.216559 3027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 03:00:08.220569 3041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:08.222046 3027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:00:08.222198 3027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/wal
uuid: "155dcc5296f545d09b5b4ca12d35d6ba"
format_stamp: "Formatted at 2026-09-29 03:00:08 on dist-test-slave-q5f0"
server_key: "cfc5262e1cc1f896829c2b90d7d7b09a"
server_key_iv: "b49e6ab41ce75eb993fbb83f125632d4"
server_key_version: "encryptionkey@0"
I20260929 03:00:08.222416 3027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:08.239733 3027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:08.240614 3027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:08.240908 3027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:08.241951 3027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:00:08.243541 3027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:00:08.243647 3027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:08.243759 3027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:00:08.243832 3027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:08.276103 3027 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.66:33693
I20260929 03:00:08.276109 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.66:33693 every 8 connection(s)
I20260929 03:00:08.277439 3027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/data/info.pb
I20260929 03:00:08.285966 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 3027
I20260929 03:00:08.286128 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/wal/instance
I20260929 03:00:08.286553 15433 external_mini_cluster.cc:1428] Setting key e5ef0c0436ebd2bca8b601bafdfd9ab0
I20260929 03:00:08.289353 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.18.67:0
--local_ip_for_outbound_sockets=127.15.18.67
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39443
--builtin_ntp_servers=127.15.18.84:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:00:08.289929 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.280091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:34261 (local address 127.15.18.126:39443)
0929 03:00:08.280303 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:08.280308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:08.281661 (+ 1353us) server_negotiation.cc:407] Connection header received
0929 03:00:08.283455 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:08.283460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:08.283581 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:08.283669 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:08.284759 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.285489 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.287134 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.287730 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.287843 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.289138 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:08.289142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:08.289145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:08.289162 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:08.289204 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:08.289217 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:08.289289 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:08.289533 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 03:00:08.289630 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 03:00:08.290963 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.280543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:39443 (local address 127.15.18.66:34261)
0929 03:00:08.281476 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:08.281513 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:08.283176 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:08.283930 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:08.283939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:08.284032 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:08.284589 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:08.284605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.285648 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.285651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:08.286994 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:08.287006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.287949 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.287957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:08.288169 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.288821 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:00:08.288823 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:08.288865 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:00:08.288945 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:08.289310 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:08.289320 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:08.289327 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:00:08.289503 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 03:00:08.289933 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":303,"threads_started":1}
I20260929 03:00:08.292481 3155 heartbeater.cc:347] Connected to a master server at 127.15.18.126:39443
I20260929 03:00:08.292807 3155 heartbeater.cc:464] Registering TS with master...
I20260929 03:00:08.293696 3155 heartbeater.cc:511] Master 127.15.18.126:39443 requested a full tablet report, sending...
I20260929 03:00:08.295291 2835 ts_manager.cc:194] Registered new tserver with Master: 155dcc5296f545d09b5b4ca12d35d6ba (127.15.18.66:33693)
I20260929 03:00:08.296319 2835 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.66:34261
W20260929 03:00:08.619356 3158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:08.619656 3158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:08.619742 3158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:08.626984 3158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:08.627089 3158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:08.627251 3158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.18.67
I20260929 03:00:08.636512 3158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.18.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.18.67
--webserver_port=0
--tserver_master_addrs=127.15.18.126:39443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.18.67
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:08.637933 3158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:08.639791 3158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:08.652834 3164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:08.653036 3166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:08.652853 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:08.653715 3158 server_base.cc:1034] running on GCE node
I20260929 03:00:08.654402 3158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:08.655459 3158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:08.656751 3158 hybrid_clock.cc:648] HybridClock initialized: now 1790650808656676 us; error 35 us; skew 500 ppm
I20260929 03:00:08.657110 3158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 03:00:08.659595 3158 webserver.cc:460] Webserver started at http://127.15.18.67:37247/ using document root <none> and password file <none>
I20260929 03:00:08.660373 3158 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:08.660481 3158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:08.660826 3158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:08.664011 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/data/instance:
uuid: "4df34e7fd20448c0bf6e1720c6493cda"
format_stamp: "Formatted at 2026-09-29 03:00:08 on dist-test-slave-q5f0"
server_key: "c3f903eddfc0b7d3027cff181ad5422e"
server_key_iv: "b7bc5f8c9bcc03bc4ada2831abec63ed"
server_key_version: "encryptionkey@0"
I20260929 03:00:08.664856 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance:
uuid: "4df34e7fd20448c0bf6e1720c6493cda"
format_stamp: "Formatted at 2026-09-29 03:00:08 on dist-test-slave-q5f0"
server_key: "c3f903eddfc0b7d3027cff181ad5422e"
server_key_iv: "b7bc5f8c9bcc03bc4ada2831abec63ed"
server_key_version: "encryptionkey@0"
I20260929 03:00:08.670298 3158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:00:08.673895 3172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:08.675370 3158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:00:08.675536 3158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/wal
uuid: "4df34e7fd20448c0bf6e1720c6493cda"
format_stamp: "Formatted at 2026-09-29 03:00:08 on dist-test-slave-q5f0"
server_key: "c3f903eddfc0b7d3027cff181ad5422e"
server_key_iv: "b7bc5f8c9bcc03bc4ada2831abec63ed"
server_key_version: "encryptionkey@0"
I20260929 03:00:08.675791 3158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:08.709260 3158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:08.710193 3158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:08.710490 3158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:08.711557 3158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:00:08.713119 3158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:00:08.713208 3158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:08.713322 3158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:00:08.713385 3158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:08.745285 3158 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.67:46109
I20260929 03:00:08.745368 3284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.67:46109 every 8 connection(s)
I20260929 03:00:08.746620 3158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/data/info.pb
I20260929 03:00:08.756347 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 3158
I20260929 03:00:08.756495 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/wal/instance
I20260929 03:00:08.757054 15433 external_mini_cluster.cc:1428] Setting key e9d329c7f5ea9df92856d53230ff6804
I20260929 03:00:08.759904 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.749374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.67:59347 (local address 127.15.18.126:39443)
0929 03:00:08.749564 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:08.749569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:08.751033 (+ 1464us) server_negotiation.cc:407] Connection header received
0929 03:00:08.752871 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:08.752876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:08.752957 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:08.753064 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:08.754182 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.755003 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.756814 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.757410 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.757552 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.759051 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:08.759057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:08.759061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:08.759082 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:08.759122 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:08.759137 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:08.759213 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:08.759481 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 03:00:08.759607 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 03:00:08.761272 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.749919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:39443 (local address 127.15.18.67:59347)
0929 03:00:08.750857 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:08.750899 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:08.752557 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:08.753284 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:08.753293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:08.753385 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:08.753997 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:08.754016 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.755166 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.755176 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:08.756654 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:08.756667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.757663 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.757672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:08.757957 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.758676 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:00:08.758680 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:08.758768 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:00:08.758879 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:08.759267 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:08.759277 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:08.759285 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:00:08.759478 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 03:00:08.759945 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":285,"threads_started":1}
I20260929 03:00:08.762434 3285 heartbeater.cc:347] Connected to a master server at 127.15.18.126:39443
I20260929 03:00:08.762689 3285 heartbeater.cc:464] Registering TS with master...
I20260929 03:00:08.763370 3285 heartbeater.cc:511] Master 127.15.18.126:39443 requested a full tablet report, sending...
I20260929 03:00:08.764807 2835 ts_manager.cc:194] Registered new tserver with Master: 4df34e7fd20448c0bf6e1720c6493cda (127.15.18.67:46109)
I20260929 03:00:08.765616 2835 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.18.67:59347
I20260929 03:00:08.772912 15433 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:00:08.783131 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.777127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.15.18.126:39443)
0929 03:00:08.777398 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:08.777405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:08.777676 (+ 271us) server_negotiation.cc:407] Connection header received
0929 03:00:08.777872 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:08.777877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:08.777947 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:08.778039 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:08.778516 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.779280 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.780739 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.781147 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.781242 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.782364 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:08.782368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:08.782371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:08.782388 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:08.782423 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:08.782435 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:08.782504 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:08.782699 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 03:00:08.782812 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 03:00:08.787061 2835 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:00:08.794226 2835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:00:08.817521 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.806519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46109 (local address 127.0.0.1:46548)
0929 03:00:08.808342 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:08.808384 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:08.808545 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:08.810513 (+ 1968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:08.810522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:08.810559 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:08.811029 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:08.811042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.813058 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.813063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:08.815134 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:08.815146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.816528 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.816534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:08.816755 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.816810 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 03:00:08.816952 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 03:00:08.817140 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1580,"mutex_wait_us":6,"thread_start_us":2669,"threads_started":1}
I20260929 03:00:08.817555 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.806840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.15.18.67:46109)
0929 03:00:08.809196 (+ 2356us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:08.809205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:08.809239 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:00:08.809950 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:08.809961 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:08.810249 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:08.811240 (+ 991us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:08.811551 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.812886 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.815342 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.816322 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.816443 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.816727 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:08.817013 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 03:00:08.817182 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2166,"thread_start_us":1998,"threads_started":1}
I20260929 03:00:08.819240 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.806679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:40425 (local address 127.0.0.1:45926)
0929 03:00:08.809480 (+ 2801us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:08.809498 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:08.809605 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:08.812377 (+ 2772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:08.812382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:08.812409 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:08.812778 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:08.812790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.814574 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.814579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:08.816708 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:08.816721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.818437 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.818444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:08.818616 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.818647 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:00:08.818741 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 03:00:08.818865 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2648,"mutex_wait_us":34,"thread_start_us":1290,"threads_started":1}
I20260929 03:00:08.819278 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.807264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.15.18.65:40425)
0929 03:00:08.811839 (+ 4575us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:08.811847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:08.811874 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:00:08.811964 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:08.811971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:08.812187 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:08.812362 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:08.812945 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.814428 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.816890 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.818240 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.818371 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.818596 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:08.818797 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 03:00:08.818953 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4408,"thread_start_us":4299,"threads_started":1}
I20260929 03:00:08.819979 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.810174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.66:33693 (local address 127.0.0.1:42430)
0929 03:00:08.810860 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:08.810888 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:08.811010 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:08.812087 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:08.812092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:08.812119 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:08.812458 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:08.812475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.814162 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.814167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:08.816396 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:08.816410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.819222 (+ 2812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.819230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:08.819418 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.819450 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:00:08.819551 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:00:08.819672 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":54,"thread_start_us":425,"threads_started":1}
I20260929 03:00:08.820077 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:08.809789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.15.18.66:33693)
0929 03:00:08.811456 (+ 1667us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:08.811474 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:08.811505 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:00:08.811621 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:08.811629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:08.811876 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:08.812063 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:08.812681 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.814027 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.816616 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:08.818766 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:08.818887 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:08.819097 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:08.819609 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 03:00:08.819770 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1536,"thread_start_us":1426,"threads_started":1}
I20260929 03:00:08.821775 3220 tablet_service.cc:1467] Processing CreateTablet for tablet 9d86ef67e7f947a99ca7872d72473495 (DEFAULT_TABLE table=test-table [id=bbdc28848e9246c780148bd74389c816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:00:08.822427 2957 tablet_service.cc:1467] Processing CreateTablet for tablet 9d86ef67e7f947a99ca7872d72473495 (DEFAULT_TABLE table=test-table [id=bbdc28848e9246c780148bd74389c816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:00:08.823709 3220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d86ef67e7f947a99ca7872d72473495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:08.823760 2957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d86ef67e7f947a99ca7872d72473495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:08.825989 3089 tablet_service.cc:1467] Processing CreateTablet for tablet 9d86ef67e7f947a99ca7872d72473495 (DEFAULT_TABLE table=test-table [id=bbdc28848e9246c780148bd74389c816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:00:08.827358 3089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d86ef67e7f947a99ca7872d72473495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:08.838058 3303 tablet_bootstrap.cc:492] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: Bootstrap starting.
I20260929 03:00:08.841482 3304 tablet_bootstrap.cc:492] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: Bootstrap starting.
I20260929 03:00:08.841670 3303 tablet_bootstrap.cc:654] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:08.842864 3305 tablet_bootstrap.cc:492] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: Bootstrap starting.
I20260929 03:00:08.842949 3303 log.cc:824] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:08.844875 3304 tablet_bootstrap.cc:654] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:08.845938 3303 tablet_bootstrap.cc:492] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: No bootstrap required, opened a new log
I20260929 03:00:08.846052 3304 log.cc:824] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:08.846197 3303 ts_tablet_manager.cc:1397] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 03:00:08.846460 3305 tablet_bootstrap.cc:654] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:08.847497 3305 log.cc:824] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:08.849067 3304 tablet_bootstrap.cc:492] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: No bootstrap required, opened a new log
I20260929 03:00:08.849334 3304 ts_tablet_manager.cc:1397] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260929 03:00:08.850425 3305 tablet_bootstrap.cc:492] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: No bootstrap required, opened a new log
I20260929 03:00:08.850677 3305 ts_tablet_manager.cc:1397] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 03:00:08.854180 3303 raft_consensus.cc:348] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 } } peers { permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 } } peers { permanent_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 33693 } }
I20260929 03:00:08.854565 3303 raft_consensus.cc:374] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:08.854661 3303 raft_consensus.cc:729] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df34e7fd20448c0bf6e1720c6493cda, State: Initialized, Role: FOLLOWER
I20260929 03:00:08.855257 3303 consensus_queue.cc:260] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 } } peers { permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 } } peers { permanent_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 33693 } }
I20260929 03:00:08.856953 3023 heartbeater.cc:503] Master 127.15.18.126:39443 was elected leader, sending a full tablet report...
I20260929 03:00:08.859020 3285 heartbeater.cc:503] Master 127.15.18.126:39443 was elected leader, sending a full tablet report...
I20260929 03:00:08.859409 3304 raft_consensus.cc:348] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 } } peers { permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 } } peers { permanent_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 33693 } }
I20260929 03:00:08.859925 3304 raft_consensus.cc:374] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:08.860030 3304 raft_consensus.cc:729] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155dcc5296f545d09b5b4ca12d35d6ba, State: Initialized, Role: FOLLOWER
I20260929 03:00:08.860424 3303 ts_tablet_manager.cc:1428] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260929 03:00:08.860737 3304 consensus_queue.cc:260] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 } } peers { permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 } } peers { permanent_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 33693 } }
I20260929 03:00:08.862826 3305 raft_consensus.cc:348] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 } } peers { permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 } } peers { permanent_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 33693 } }
I20260929 03:00:08.863222 3305 raft_consensus.cc:374] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:08.863317 3305 raft_consensus.cc:729] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f6097d4ffe482cbc461f1bb0d5ee74, State: Initialized, Role: FOLLOWER
I20260929 03:00:08.863906 3305 consensus_queue.cc:260] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 } } peers { permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 } } peers { permanent_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 33693 } }
I20260929 03:00:08.865607 3305 ts_tablet_manager.cc:1428] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260929 03:00:08.866518 3155 heartbeater.cc:503] Master 127.15.18.126:39443 was elected leader, sending a full tablet report...
I20260929 03:00:08.867031 3304 ts_tablet_manager.cc:1428] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
W20260929 03:00:08.998359 3286 tablet.cc:2369] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:00:09.029443 3156 tablet.cc:2369] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:00:09.031589 3310 raft_consensus.cc:482] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:00:09.031941 3310 raft_consensus.cc:504] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 } } peers { permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 } } peers { permanent_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 33693 } }
I20260929 03:00:09.033655 3310 leader_election.cc:288] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4df34e7fd20448c0bf6e1720c6493cda (127.15.18.67:46109), b7f6097d4ffe482cbc461f1bb0d5ee74 (127.15.18.65:40425)
I20260929 03:00:09.040450 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:09.034100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:43339 (local address 127.15.18.67:46109)
0929 03:00:09.034295 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:09.034301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:09.035185 (+ 884us) server_negotiation.cc:407] Connection header received
0929 03:00:09.035365 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:09.035370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:09.035435 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:09.035535 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:09.036171 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.036890 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.038792 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.039610 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.039777 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:09.039847 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:09.040097 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 03:00:09.040222 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:00:09.040467 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:09.034100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.18.66:42817 (local address 127.15.18.65:40425)
0929 03:00:09.034354 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:09.034361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:09.035083 (+ 722us) server_negotiation.cc:407] Connection header received
0929 03:00:09.035278 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:09.035282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:09.035352 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:09.035427 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:00:09.035995 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.036708 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.038795 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.039566 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.039705 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:09.039775 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:09.040089 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 03:00:09.040218 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 03:00:09.040601 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:09.034148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.65:40425 (local address 127.15.18.66:42817)
0929 03:00:09.034961 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:09.034978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:09.035124 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:09.035468 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:09.035473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:09.035495 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:09.035849 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:09.035859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.036850 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.036853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:09.038562 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:09.038575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.039775 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.039782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:09.039929 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:09.039954 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 03:00:09.040034 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:00:09.040160 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":126,"thread_start_us":442,"threads_started":1}
I20260929 03:00:09.040863 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:09.034143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.67:46109 (local address 127.15.18.66:43339)
0929 03:00:09.035061 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:09.035076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:09.035212 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:09.035581 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:09.035585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:09.035624 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:00:09.036022 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:09.036031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.037029 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.037035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:09.038565 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:09.038577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.039782 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.039789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:09.039951 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:09.039974 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:00:09.040051 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:00:09.040171 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":77,"thread_start_us":412,"threads_started":1}
I20260929 03:00:09.041699 2977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d86ef67e7f947a99ca7872d72473495" candidate_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" is_pre_election: true
I20260929 03:00:09.041920 3240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d86ef67e7f947a99ca7872d72473495" candidate_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df34e7fd20448c0bf6e1720c6493cda" is_pre_election: true
I20260929 03:00:09.042344 2977 raft_consensus.cc:2454] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155dcc5296f545d09b5b4ca12d35d6ba in term 0.
I20260929 03:00:09.042382 3240 raft_consensus.cc:2454] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155dcc5296f545d09b5b4ca12d35d6ba in term 0.
I20260929 03:00:09.043162 3043 leader_election.cc:302] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 155dcc5296f545d09b5b4ca12d35d6ba, 4df34e7fd20448c0bf6e1720c6493cda, b7f6097d4ffe482cbc461f1bb0d5ee74; no voters:
I20260929 03:00:09.043643 3310 raft_consensus.cc:2781] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:00:09.043778 3310 raft_consensus.cc:482] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:00:09.043972 3310 raft_consensus.cc:3037] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:09.046131 3310 raft_consensus.cc:504] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 } } peers { permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 } } peers { permanent_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 33693 } }
I20260929 03:00:09.046767 3310 leader_election.cc:288] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [CANDIDATE]: Term 1 election: Requested vote from peers 4df34e7fd20448c0bf6e1720c6493cda (127.15.18.67:46109), b7f6097d4ffe482cbc461f1bb0d5ee74 (127.15.18.65:40425)
I20260929 03:00:09.047384 3240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d86ef67e7f947a99ca7872d72473495" candidate_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df34e7fd20448c0bf6e1720c6493cda"
I20260929 03:00:09.047408 2977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d86ef67e7f947a99ca7872d72473495" candidate_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74"
I20260929 03:00:09.047657 2977 raft_consensus.cc:3037] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:09.047706 3240 raft_consensus.cc:3037] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:09.050925 3240 raft_consensus.cc:2454] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155dcc5296f545d09b5b4ca12d35d6ba in term 1.
I20260929 03:00:09.050917 2977 raft_consensus.cc:2454] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155dcc5296f545d09b5b4ca12d35d6ba in term 1.
I20260929 03:00:09.051441 3043 leader_election.cc:302] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 155dcc5296f545d09b5b4ca12d35d6ba, 4df34e7fd20448c0bf6e1720c6493cda, b7f6097d4ffe482cbc461f1bb0d5ee74; no voters:
I20260929 03:00:09.051800 3310 raft_consensus.cc:2781] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:09.052214 3310 raft_consensus.cc:686] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 1 LEADER]: Becoming Leader. State: Replica: 155dcc5296f545d09b5b4ca12d35d6ba, State: Running, Role: LEADER
I20260929 03:00:09.052757 3310 consensus_queue.cc:237] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 } } peers { permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 } } peers { permanent_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 33693 } }
I20260929 03:00:09.057138 2835 catalog_manager.cc:5495] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 155dcc5296f545d09b5b4ca12d35d6ba (127.15.18.66). New cstate: current_term: 1 leader_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "155dcc5296f545d09b5b4ca12d35d6ba" member_type: VOTER last_known_addr { host: "127.15.18.66" port: 33693 } health_report { overall_health: HEALTHY } } }
W20260929 03:00:09.082660 3024 tablet.cc:2369] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:00:09.112694 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:09.107294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.15.18.66:33693)
0929 03:00:09.107564 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:09.107569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:09.107589 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:00:09.107766 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:09.107770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:09.107852 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:09.107980 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:09.108345 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.109083 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.110474 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.110912 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.111031 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:09.111547 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:09.112103 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:09.112229 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:09.112327 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 03:00:09.112414 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 03:00:09.121109 2977 raft_consensus.cc:1261] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74 [term 1 FOLLOWER]: Refusing update from remote peer 155dcc5296f545d09b5b4ca12d35d6ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:00:09.121110 3240 raft_consensus.cc:1261] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda [term 1 FOLLOWER]: Refusing update from remote peer 155dcc5296f545d09b5b4ca12d35d6ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:00:09.122066 3314 consensus_queue.cc:1035] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df34e7fd20448c0bf6e1720c6493cda" member_type: VOTER last_known_addr { host: "127.15.18.67" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:00:09.122442 3310 consensus_queue.cc:1035] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f6097d4ffe482cbc461f1bb0d5ee74" member_type: VOTER last_known_addr { host: "127.15.18.65" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:00:09.133080 3317 mvcc.cc:205] Tried to move back new op lower bound from 7334505714150203392 to 7334505713883762688. Current Snapshot: MvccSnapshot[applied={T|T < 7334505714150203392}]
I20260929 03:00:09.147213 2834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:00:09.147497 2834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:00:09.151173 2834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 941dffae6bba4cc3812e513cfa78adbf: Sending DeleteTablet for 3 replicas of tablet 9d86ef67e7f947a99ca7872d72473495
I20260929 03:00:09.152508 3220 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d86ef67e7f947a99ca7872d72473495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:00:09 UTC) from {username='slave'} at 127.0.0.1:46548
I20260929 03:00:09.152596 2957 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d86ef67e7f947a99ca7872d72473495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:00:09 UTC) from {username='slave'} at 127.0.0.1:45926
I20260929 03:00:09.152696 3089 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d86ef67e7f947a99ca7872d72473495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:00:09 UTC) from {username='slave'} at 127.0.0.1:42430
I20260929 03:00:09.153448 3326 tablet_replica.cc:330] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: stopping tablet replica
I20260929 03:00:09.153993 3326 raft_consensus.cc:2229] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:00:09.154179 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 2895
I20260929 03:00:09.154668 3326 raft_consensus.cc:2258] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:00:09.161834 3326 ts_tablet_manager.cc:1905] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:00:09.164780 3328 tablet_replica.cc:330] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: stopping tablet replica
I20260929 03:00:09.165343 3328 raft_consensus.cc:2229] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:00:09.166102 3328 raft_consensus.cc:2258] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:00:09.168656 3328 ts_tablet_manager.cc:1905] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:00:09.172696 3326 ts_tablet_manager.cc:1918] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:00:09.172919 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:09.155405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.15.18.65:40425)
0929 03:00:09.155622 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:09.155636 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:09.164370 (+ 8734us) server_negotiation.cc:407] Connection header received
0929 03:00:09.164683 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:09.164689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:09.164802 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:09.164895 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:09.165805 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.166862 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.168730 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.169368 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.169484 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:09.171025 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:09.171037 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:09.171054 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:09.171846 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:09.171924 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:09.171948 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:09.172036 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:09.172278 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 03:00:09.172506 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:00:09.173556 3326 log.cc:1196] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-0/wal/wals/9d86ef67e7f947a99ca7872d72473495
I20260929 03:00:09.174145 3326 ts_tablet_manager.cc:1939] T 9d86ef67e7f947a99ca7872d72473495 P b7f6097d4ffe482cbc461f1bb0d5ee74: Deleting consensus metadata
I20260929 03:00:09.174165 3017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:09.176551 3327 tablet_replica.cc:330] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: stopping tablet replica
I20260929 03:00:09.177023 3327 raft_consensus.cc:2229] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:00:09.177556 3327 raft_consensus.cc:2258] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:00:09.179729 3328 ts_tablet_manager.cc:1918] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:00:09.179962 3328 log.cc:1196] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-1/wal/wals/9d86ef67e7f947a99ca7872d72473495
I20260929 03:00:09.180630 3328 ts_tablet_manager.cc:1939] T 9d86ef67e7f947a99ca7872d72473495 P 155dcc5296f545d09b5b4ca12d35d6ba: Deleting consensus metadata
I20260929 03:00:09.181049 3327 ts_tablet_manager.cc:1905] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:00:09.182626 2820 catalog_manager.cc:4841] TS 155dcc5296f545d09b5b4ca12d35d6ba (127.15.18.66:33693): tablet 9d86ef67e7f947a99ca7872d72473495 (table test-table [id=bbdc28848e9246c780148bd74389c816]) successfully deleted
I20260929 03:00:09.188591 3327 ts_tablet_manager.cc:1918] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:00:09.188812 3327 log.cc:1196] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: Deleting WAL directory at /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790650496826590-15433-0/minicluster-data/ts-2/wal/wals/9d86ef67e7f947a99ca7872d72473495
I20260929 03:00:09.189340 3327 ts_tablet_manager.cc:1939] T 9d86ef67e7f947a99ca7872d72473495 P 4df34e7fd20448c0bf6e1720c6493cda: Deleting consensus metadata
I20260929 03:00:09.191576 2820 catalog_manager.cc:4841] TS 4df34e7fd20448c0bf6e1720c6493cda (127.15.18.67:46109): tablet 9d86ef67e7f947a99ca7872d72473495 (table test-table [id=bbdc28848e9246c780148bd74389c816]) successfully deleted
I20260929 03:00:09.268343 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 2895
I20260929 03:00:09.268478 2822 catalog_manager.cc:4841] TS b7f6097d4ffe482cbc461f1bb0d5ee74 (127.15.18.65:40425): tablet 9d86ef67e7f947a99ca7872d72473495 (table test-table [id=bbdc28848e9246c780148bd74389c816]) successfully deleted
I20260929 03:00:09.287261 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3027
I20260929 03:00:09.297453 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:09.288391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.15.18.66:33693)
0929 03:00:09.288561 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:09.288567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:09.288594 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:00:09.288662 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:09.288668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:09.288768 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:09.288875 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:09.289355 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.290582 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.293082 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.293800 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.293932 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:09.295325 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:09.295336 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:09.295355 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:09.296224 (+ 869us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:09.296297 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:09.296322 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:09.296439 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:09.296938 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 03:00:09.297045 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:00:09.299013 3149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:09.410483 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 3027
I20260929 03:00:09.431350 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3158
I20260929 03:00:09.440340 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:09.432249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.15.18.67:46109)
0929 03:00:09.432410 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:09.432414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:09.432432 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:00:09.432617 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:09.432623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:09.432733 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:09.432850 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:09.433768 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.434877 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.436809 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:09.437437 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:09.437576 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:09.438742 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:09.438756 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:09.438777 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:09.439388 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:09.439442 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:09.439462 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:09.439771 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:09.439880 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:00:09.439999 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:00:09.441033 3280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:09.530544 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 3158
I20260929 03:00:09.543661 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 2804
I20260929 03:00:09.544889 2865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:09.654585 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 2804
2026-09-29T03:00:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2950 ms)
[----------] 6 tests from EncryptionPolicyTest (21882 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:09 dist-test-slave-q5f0 krb5kdc[3336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:09 dist-test-slave-q5f0 krb5kdc[3336](info): set up 2 sockets
Sep 29 03:00:09 dist-test-slave-q5f0 krb5kdc[3336](info): commencing operation
krb5kdc: starting...
W20260929 03:00:10.684088 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.009s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:10 dist-test-slave-q5f0 krb5kdc[3336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:10.935757 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:11.266713 3349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:11.267047 3349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:11.267171 3349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:11.274710 3349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:11.274842 3349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:11.274904 3349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:11.274989 3349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:11.275044 3349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:11.285169 3349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:11.286549 3349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:11.287889 3349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:11.300086 3354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:11.300058 3357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:11.300012 3355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:11.300608 3349 server_base.cc:1034] running on GCE node
I20260929 03:00:11.301472 3349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:11.302949 3349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:11.304140 3349 hybrid_clock.cc:648] HybridClock initialized: now 1790650811304086 us; error 51 us; skew 500 ppm
I20260929 03:00:11.304461 3349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:11 dist-test-slave-q5f0 krb5kdc[3336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:11.307713 3349 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:11.309275 3349 webserver.cc:460] Webserver started at http://127.0.0.1:39235/ using document root <none> and password file <none>
I20260929 03:00:11.309990 3349 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:11.310112 3349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:11.310415 3349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:11.313633 3349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "eb6c677691384de8884f7d9c8eb73340"
format_stamp: "Formatted at 2026-09-29 03:00:11 on dist-test-slave-q5f0"
server_key: "2c63d0ef32f27c6f93b162f696f0791b"
server_key_iv: "76df8572ac68c1d71b8ab4fff4e45b8b"
server_key_version: "encryptionkey@0"
I20260929 03:00:11.314420 3349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "eb6c677691384de8884f7d9c8eb73340"
format_stamp: "Formatted at 2026-09-29 03:00:11 on dist-test-slave-q5f0"
server_key: "2c63d0ef32f27c6f93b162f696f0791b"
server_key_iv: "76df8572ac68c1d71b8ab4fff4e45b8b"
server_key_version: "encryptionkey@0"
I20260929 03:00:11.319875 3349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260929 03:00:11.323429 3364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:11.324927 3349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:00:11.325119 3349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "eb6c677691384de8884f7d9c8eb73340"
format_stamp: "Formatted at 2026-09-29 03:00:11 on dist-test-slave-q5f0"
server_key: "2c63d0ef32f27c6f93b162f696f0791b"
server_key_iv: "76df8572ac68c1d71b8ab4fff4e45b8b"
server_key_version: "encryptionkey@0"
I20260929 03:00:11.325317 3349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:11.339561 3349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:11.344163 3349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:11.344451 3349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:11.363445 3349 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42419
I20260929 03:00:11.363461 3415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42419 every 8 connection(s)
I20260929 03:00:11.364975 3349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:11.369016 3416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:11.370545 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 3349
I20260929 03:00:11.370743 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:11.371435 15433 external_mini_cluster.cc:1428] Setting key 0649fac518d85645b99b48dcbcda5331
I20260929 03:00:11.381453 3416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340: Bootstrap starting.
I20260929 03:00:11.385860 3416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:11.387280 3416 log.cc:824] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:00:11 dist-test-slave-q5f0 krb5kdc[3336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:00:11.390659 3416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340: No bootstrap required, opened a new log
I20260929 03:00:11.398723 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:11.373447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59897 (local address 127.0.0.1:42419)
0929 03:00:11.374244 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:11.374257 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:11.374312 (+ 55us) server_negotiation.cc:407] Connection header received
0929 03:00:11.376100 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:11.376125 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:11.376558 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:11.376903 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:11.378709 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:11.380192 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:11.382163 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:11.383373 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:11.383607 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:11.389126 (+ 5519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:11.389134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:11.389160 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:11.389210 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:11.392454 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:11.394039 (+ 1585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:11.394053 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:11.394061 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:11.394281 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:11.394863 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:11.394867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:11.394870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:11.396057 (+ 1187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:11.396422 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:11.397206 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 03:00:11.397573 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":361,"threads_started":1}
I20260929 03:00:11.399557 3416 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6c677691384de8884f7d9c8eb73340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260929 03:00:11.400008 3416 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:11.400125 3416 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6c677691384de8884f7d9c8eb73340, State: Initialized, Role: FOLLOWER
I20260929 03:00:11.400835 3416 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6c677691384de8884f7d9c8eb73340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260929 03:00:11.401129 3416 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:11.401284 3416 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:11.401458 3416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:11.403394 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:11.404284 3416 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6c677691384de8884f7d9c8eb73340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260929 03:00:11.404841 3416 leader_election.cc:302] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6c677691384de8884f7d9c8eb73340; no voters:
I20260929 03:00:11.405478 3416 leader_election.cc:288] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:11.405663 3421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:11.407733 3416 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:11.408205 3421 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [term 1 LEADER]: Becoming Leader. State: Replica: eb6c677691384de8884f7d9c8eb73340, State: Running, Role: LEADER
I20260929 03:00:11.408960 3421 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6c677691384de8884f7d9c8eb73340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260929 03:00:11.412761 3422 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb6c677691384de8884f7d9c8eb73340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6c677691384de8884f7d9c8eb73340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } } }
I20260929 03:00:11.413250 3422 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:11.413753 3423 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6c677691384de8884f7d9c8eb73340. Latest consensus state: current_term: 1 leader_uuid: "eb6c677691384de8884f7d9c8eb73340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6c677691384de8884f7d9c8eb73340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } } }
I20260929 03:00:11.414054 3423 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:11.428421 3431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:11.434943 3431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:11.446399 3431 catalog_manager.cc:1347] Generated new cluster ID: 12591490bcca429ebe10568b58126c46
I20260929 03:00:11.446516 3431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:11.448319 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:11.419757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47055 (local address 127.0.0.1:42419)
0929 03:00:11.419966 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:11.419974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:11.427311 (+ 7337us) server_negotiation.cc:407] Connection header received
0929 03:00:11.427908 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:11.427914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:11.428004 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:11.428083 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:11.430470 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:11.431655 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:11.433413 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:11.434106 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:11.434256 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:11.437434 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:11.437440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:11.437445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:11.437490 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:11.444253 (+ 6763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:11.445332 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:11.445337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:11.445340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:11.445503 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:11.446071 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:11.446077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:11.446080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:11.446625 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:11.446884 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:11.447778 (+ 894us) server_negotiation.cc:299] Negotiation successful
0929 03:00:11.447973 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:00:11.460781 3431 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:11.462657 3431 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:11.475539 3431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6c677691384de8884f7d9c8eb73340: Generated new TSK 0
I20260929 03:00:11.476430 3431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:11.486665 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3349
I20260929 03:00:11.488024 3411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:11.578727 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 3349
W20260929 03:00:11.592844 3334 connection.cc:535] client connection to 127.0.0.1:42419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42419): Connection reset by peer (error 104)
2026-09-29T03:00:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:11 dist-test-slave-q5f0 krb5kdc[3448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:11 dist-test-slave-q5f0 krb5kdc[3448](info): set up 2 sockets
Sep 29 03:00:11 dist-test-slave-q5f0 krb5kdc[3448](info): commencing operation
krb5kdc: starting...
W20260929 03:00:12.596141 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:12 dist-test-slave-q5f0 krb5kdc[3448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:12.846724 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:13.175246 3461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:13.175617 3461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:13.175784 3461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:13.183382 3461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:13.183538 3461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:13.183609 3461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:13.183694 3461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:13.183759 3461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:13.194298 3461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:13.195688 3461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:13.197113 3461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:13.209326 3467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:13.209419 3466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:13.209385 3469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:13.210175 3461 server_base.cc:1034] running on GCE node
I20260929 03:00:13.211087 3461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:13.212754 3461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:13.214000 3461 hybrid_clock.cc:648] HybridClock initialized: now 1790650813213943 us; error 53 us; skew 500 ppm
I20260929 03:00:13.214396 3461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:13 dist-test-slave-q5f0 krb5kdc[3448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:13.221801 3461 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:13.223755 3461 webserver.cc:460] Webserver started at http://127.0.0.1:34763/ using document root <none> and password file <none>
I20260929 03:00:13.224489 3461 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:13.224622 3461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:13.224936 3461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:13.228175 3461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "9256ec6a18c34ef885b58ed0542198b9"
format_stamp: "Formatted at 2026-09-29 03:00:13 on dist-test-slave-q5f0"
server_key: "ab4c3e9233a197823646f97863a2539d"
server_key_iv: "f50d6817273992b9eb7bd464a9a9ca9a"
server_key_version: "encryptionkey@0"
I20260929 03:00:13.228982 3461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "9256ec6a18c34ef885b58ed0542198b9"
format_stamp: "Formatted at 2026-09-29 03:00:13 on dist-test-slave-q5f0"
server_key: "ab4c3e9233a197823646f97863a2539d"
server_key_iv: "f50d6817273992b9eb7bd464a9a9ca9a"
server_key_version: "encryptionkey@0"
I20260929 03:00:13.234401 3461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:00:13.238169 3476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:13.239781 3461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 03:00:13.240026 3461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "9256ec6a18c34ef885b58ed0542198b9"
format_stamp: "Formatted at 2026-09-29 03:00:13 on dist-test-slave-q5f0"
server_key: "ab4c3e9233a197823646f97863a2539d"
server_key_iv: "f50d6817273992b9eb7bd464a9a9ca9a"
server_key_version: "encryptionkey@0"
I20260929 03:00:13.240226 3461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:13.273422 3461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:13.278064 3461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:13.278349 3461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:13.296537 3461 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41679
I20260929 03:00:13.296597 3527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41679 every 8 connection(s)
I20260929 03:00:13.298008 3461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:13.301321 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 3461
I20260929 03:00:13.301527 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:13.302124 15433 external_mini_cluster.cc:1428] Setting key 816614b8198bbda81c6cd352498879b7
I20260929 03:00:13.302639 3528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:13.316533 3528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9: Bootstrap starting.
I20260929 03:00:13.316915 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:13.304237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45611 (local address 127.0.0.1:41679)
0929 03:00:13.305026 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:13.305039 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:13.305081 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:00:13.307412 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:13.307434 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:13.307941 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:13.308300 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:13.308921 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:13.310362 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:13.312179 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:13.312909 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:13.313059 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:13.314359 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:13.314367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:13.314386 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:13.314957 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:13.315037 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:13.315063 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:13.315182 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:13.315577 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 03:00:13.315796 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":367,"threads_started":1}
I20260929 03:00:13.319814 3528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:13.321097 3528 log.cc:824] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:13.321337 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:13.324193 3528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9: No bootstrap required, opened a new log
I20260929 03:00:13.331977 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:13.325894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43615 (local address 127.0.0.1:41679)
0929 03:00:13.326056 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:13.326062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:13.326481 (+ 419us) server_negotiation.cc:407] Connection header received
0929 03:00:13.326677 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:13.326681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:13.326750 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:13.326838 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:13.327396 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:13.328234 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:13.329557 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:13.330016 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:13.330122 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:13.331266 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:13.331270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:13.331274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:13.331291 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:13.331337 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:13.331352 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:13.331445 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:13.331657 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 03:00:13.331712 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:00:13.332213 3528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ec6a18c34ef885b58ed0542198b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260929 03:00:13.332770 3528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:13.332885 3528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9256ec6a18c34ef885b58ed0542198b9, State: Initialized, Role: FOLLOWER
I20260929 03:00:13.333653 3528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ec6a18c34ef885b58ed0542198b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260929 03:00:13.333937 3528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:13.334043 3528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:13.334177 3528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:13.336457 3528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ec6a18c34ef885b58ed0542198b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260929 03:00:13.336999 3528 leader_election.cc:302] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9256ec6a18c34ef885b58ed0542198b9; no voters:
I20260929 03:00:13.337704 3528 leader_election.cc:288] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:13.337852 3538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:13.338534 3538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [term 1 LEADER]: Becoming Leader. State: Replica: 9256ec6a18c34ef885b58ed0542198b9, State: Running, Role: LEADER
I20260929 03:00:13.339191 3538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ec6a18c34ef885b58ed0542198b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260929 03:00:13.341166 3528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:13.342388 3540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9256ec6a18c34ef885b58ed0542198b9. Latest consensus state: current_term: 1 leader_uuid: "9256ec6a18c34ef885b58ed0542198b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ec6a18c34ef885b58ed0542198b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } } }
I20260929 03:00:13.342360 3539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9256ec6a18c34ef885b58ed0542198b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9256ec6a18c34ef885b58ed0542198b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } } }
I20260929 03:00:13.342806 3540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:13.342806 3539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:13.356894 3543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:13.361342 3543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:13.370153 3543 catalog_manager.cc:1347] Generated new cluster ID: 95a8ffe2313d428392f549a1b05574f8
I20260929 03:00:13.370285 3543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:13.393260 3543 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:13.395155 3543 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:13.407832 3543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9256ec6a18c34ef885b58ed0542198b9: Generated new TSK 0
I20260929 03:00:13.408752 3543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:13.421746 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3461
W20260929 03:00:13.423120 3523 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45611
W20260929 03:00:13.424197 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3461: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:13.424403 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 3461
W20260929 03:00:13.441181 3446 connection.cc:535] client connection to 127.0.0.1:41679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41679): Connection reset by peer (error 104)
2026-09-29T03:00:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1848 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:13 dist-test-slave-q5f0 krb5kdc[3559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:13 dist-test-slave-q5f0 krb5kdc[3559](info): set up 2 sockets
Sep 29 03:00:13 dist-test-slave-q5f0 krb5kdc[3559](info): commencing operation
krb5kdc: starting...
W20260929 03:00:14.456077 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:14 dist-test-slave-q5f0 krb5kdc[3559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:14.712711 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:15.032102 3572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:15.032501 3572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:15.032655 3572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:15.040128 3572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:15.040277 3572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:15.040351 3572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:15.040437 3572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:15.040500 3572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:15.050649 3572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37493
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:15.052014 3572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:15.053328 3572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:15.065425 3577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:15.065387 3580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:15.065349 3578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:15.065627 3572 server_base.cc:1034] running on GCE node
I20260929 03:00:15.066633 3572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:15.068084 3572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:15.069356 3572 hybrid_clock.cc:648] HybridClock initialized: now 1790650815069293 us; error 44 us; skew 500 ppm
I20260929 03:00:15.069715 3572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:15 dist-test-slave-q5f0 krb5kdc[3559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:15.073158 3572 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:15.074723 3572 webserver.cc:460] Webserver started at http://127.0.0.1:34547/ using document root <none> and password file <none>
I20260929 03:00:15.075395 3572 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:15.075547 3572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:15.075871 3572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:15.078971 3572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "8056aec3ee6141ae82d5760fbe50cd41"
format_stamp: "Formatted at 2026-09-29 03:00:15 on dist-test-slave-q5f0"
server_key: "b03f060abd04f115b7c92edb07cb54af"
server_key_iv: "9f8b75d1702feb3b7ded472cc063fbad"
server_key_version: "encryptionkey@0"
I20260929 03:00:15.079808 3572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "8056aec3ee6141ae82d5760fbe50cd41"
format_stamp: "Formatted at 2026-09-29 03:00:15 on dist-test-slave-q5f0"
server_key: "b03f060abd04f115b7c92edb07cb54af"
server_key_iv: "9f8b75d1702feb3b7ded472cc063fbad"
server_key_version: "encryptionkey@0"
I20260929 03:00:15.085386 3572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 03:00:15.089154 3587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:15.090691 3572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 03:00:15.090895 3572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "8056aec3ee6141ae82d5760fbe50cd41"
format_stamp: "Formatted at 2026-09-29 03:00:15 on dist-test-slave-q5f0"
server_key: "b03f060abd04f115b7c92edb07cb54af"
server_key_iv: "9f8b75d1702feb3b7ded472cc063fbad"
server_key_version: "encryptionkey@0"
I20260929 03:00:15.091090 3572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:15.096984 3572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:15.101063 3572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:15.101305 3572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:15.118574 3572 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37493
I20260929 03:00:15.118615 3638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37493 every 8 connection(s)
I20260929 03:00:15.120301 3572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:15.124073 3639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:15.125479 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 3572
I20260929 03:00:15.125634 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:15.126076 15433 external_mini_cluster.cc:1428] Setting key 9a152c20972edb3f9de304f12de17e85
I20260929 03:00:15.133488 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:15.127776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54021 (local address 127.0.0.1:37493)
0929 03:00:15.128530 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:15.128543 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:15.128591 (+ 48us) server_negotiation.cc:407] Connection header received
0929 03:00:15.130140 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:15.130161 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:15.130528 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:15.130855 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:15.131949 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:15.131955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:15.131968 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:15.131983 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:15.132046 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:15.132068 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:15.132234 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:15.132345 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 03:00:15.132527 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":338,"threads_started":1}
I20260929 03:00:15.137279 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:15.137454 3639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41: Bootstrap starting.
I20260929 03:00:15.140403 3639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:15.141607 3639 log.cc:824] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:15.143644 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:15.141321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40193 (local address 127.0.0.1:37493)
0929 03:00:15.141479 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:15.141486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:15.141853 (+ 367us) server_negotiation.cc:407] Connection header received
0929 03:00:15.142032 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:15.142037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:15.142092 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:15.142164 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:15.142997 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:15.143003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:15.143008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:15.143026 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:15.143062 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:15.143075 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:15.143166 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:15.143272 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 03:00:15.143337 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:00:15.144407 3639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41: No bootstrap required, opened a new log
I20260929 03:00:15.151531 3639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8056aec3ee6141ae82d5760fbe50cd41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37493 } }
I20260929 03:00:15.151893 3639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:15.151980 3639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8056aec3ee6141ae82d5760fbe50cd41, State: Initialized, Role: FOLLOWER
I20260929 03:00:15.152530 3639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8056aec3ee6141ae82d5760fbe50cd41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37493 } }
I20260929 03:00:15.152730 3639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:15.152854 3639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:15.152994 3639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:15.155061 3639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8056aec3ee6141ae82d5760fbe50cd41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37493 } }
I20260929 03:00:15.155602 3639 leader_election.cc:302] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8056aec3ee6141ae82d5760fbe50cd41; no voters:
I20260929 03:00:15.156241 3639 leader_election.cc:288] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:15.156404 3649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:15.156991 3649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [term 1 LEADER]: Becoming Leader. State: Replica: 8056aec3ee6141ae82d5760fbe50cd41, State: Running, Role: LEADER
I20260929 03:00:15.157505 3649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8056aec3ee6141ae82d5760fbe50cd41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37493 } }
I20260929 03:00:15.158164 3639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:15.160547 3650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8056aec3ee6141ae82d5760fbe50cd41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8056aec3ee6141ae82d5760fbe50cd41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37493 } } }
I20260929 03:00:15.160501 3651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8056aec3ee6141ae82d5760fbe50cd41. Latest consensus state: current_term: 1 leader_uuid: "8056aec3ee6141ae82d5760fbe50cd41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8056aec3ee6141ae82d5760fbe50cd41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37493 } } }
I20260929 03:00:15.161089 3650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:15.161134 3651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41 [sys.catalog]: This master's current role is: LEADER
W20260929 03:00:15.169668 3665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:00:15.169799 3665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:00:15.169926 3656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:15.172158 3656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:15.180907 3656 catalog_manager.cc:1347] Generated new cluster ID: 990e58649c2f447993fc720ba3323eeb
I20260929 03:00:15.181039 3656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:15.214953 3656 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:15.215473 3656 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:15.224673 3656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8056aec3ee6141ae82d5760fbe50cd41: Generated new TSK 0
I20260929 03:00:15.225558 3656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:15.230989 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3572
W20260929 03:00:15.232021 3634 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54021
W20260929 03:00:15.232853 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3572: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:15.232996 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 3572
2026-09-29T03:00:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1805 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:15 dist-test-slave-q5f0 krb5kdc[3670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:15 dist-test-slave-q5f0 krb5kdc[3670](info): set up 2 sockets
Sep 29 03:00:15 dist-test-slave-q5f0 krb5kdc[3670](info): commencing operation
krb5kdc: starting...
W20260929 03:00:16.248021 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:16 dist-test-slave-q5f0 krb5kdc[3670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:16.494491 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:16.819651 3683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:16.819957 3683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:16.820097 3683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:16.827466 3683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:16.827617 3683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:16.827706 3683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:16.827783 3683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:16.827831 3683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:16.838060 3683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:16.839421 3683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:16.840737 3683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:16.852341 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:16.852399 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:16.852377 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:16.853052 3683 server_base.cc:1034] running on GCE node
I20260929 03:00:16.853920 3683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:16.855527 3683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:16.856765 3683 hybrid_clock.cc:648] HybridClock initialized: now 1790650816856738 us; error 56 us; skew 500 ppm
I20260929 03:00:16.857153 3683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:16 dist-test-slave-q5f0 krb5kdc[3670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:16.860451 3683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:16.861996 3683 webserver.cc:460] Webserver started at http://127.0.0.1:39973/ using document root <none> and password file <none>
I20260929 03:00:16.862668 3683 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:16.862789 3683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:16.863080 3683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:16.866248 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb"
format_stamp: "Formatted at 2026-09-29 03:00:16 on dist-test-slave-q5f0"
server_key: "d980f9e91f71aacec9f3a9f6775b3ae0"
server_key_iv: "ca4d848ec4bca7f72fb10992d389b936"
server_key_version: "encryptionkey@0"
I20260929 03:00:16.867007 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb"
format_stamp: "Formatted at 2026-09-29 03:00:16 on dist-test-slave-q5f0"
server_key: "d980f9e91f71aacec9f3a9f6775b3ae0"
server_key_iv: "ca4d848ec4bca7f72fb10992d389b936"
server_key_version: "encryptionkey@0"
I20260929 03:00:16.872218 3683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 03:00:16.875602 3698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:16.876894 3683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:00:16.877055 3683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb"
format_stamp: "Formatted at 2026-09-29 03:00:16 on dist-test-slave-q5f0"
server_key: "d980f9e91f71aacec9f3a9f6775b3ae0"
server_key_iv: "ca4d848ec4bca7f72fb10992d389b936"
server_key_version: "encryptionkey@0"
I20260929 03:00:16.877233 3683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:16.900825 3683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:16.905189 3683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:16.905462 3683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:16.922705 3683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35437
I20260929 03:00:16.922727 3749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35437 every 8 connection(s)
I20260929 03:00:16.924142 3683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:16.928030 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 3683
I20260929 03:00:16.928210 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:16.928223 3750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:16.928745 15433 external_mini_cluster.cc:1428] Setting key f3aad3c3355b80e4e3d983dc5d7110ca
I20260929 03:00:16.943571 3750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb: Bootstrap starting.
Sep 29 03:00:16 dist-test-slave-q5f0 krb5kdc[3670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:00:16.947902 3750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:16.949365 3750 log.cc:824] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:16.952889 3750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb: No bootstrap required, opened a new log
I20260929 03:00:16.954052 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:16.930588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43807 (local address 127.0.0.1:35437)
0929 03:00:16.931483 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:16.931495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:16.931580 (+ 85us) server_negotiation.cc:407] Connection header received
0929 03:00:16.933140 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:16.933162 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:16.933575 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:16.933928 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:16.935972 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:16.937286 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:16.939111 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:16.939921 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:16.940063 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:16.945073 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:16.945086 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:16.945107 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:16.945171 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:16.949246 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:16.950040 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:16.950054 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:16.950069 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:16.950273 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:16.950779 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:16.950784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:16.950788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:16.951480 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:16.951838 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:16.952153 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 03:00:16.952600 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":427,"threads_started":1}
I20260929 03:00:16.957844 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:16.963212 3750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35437 } }
I20260929 03:00:16.963747 3750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:16.963860 3750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd5b1cd11ef44c0b33ba4933d2d7cdb, State: Initialized, Role: FOLLOWER
I20260929 03:00:16.964654 3750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35437 } }
I20260929 03:00:16.964910 3750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:16.965068 3750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:16.965265 3750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:16.967749 3750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35437 } }
I20260929 03:00:16.968286 3750 leader_election.cc:302] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dd5b1cd11ef44c0b33ba4933d2d7cdb; no voters:
I20260929 03:00:16.968945 3750 leader_election.cc:288] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:16.969069 3760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:16.971302 3760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [term 1 LEADER]: Becoming Leader. State: Replica: 8dd5b1cd11ef44c0b33ba4933d2d7cdb, State: Running, Role: LEADER
I20260929 03:00:16.971539 3750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:16.971967 3760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35437 } }
I20260929 03:00:16.977133 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:16.963410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45067 (local address 127.0.0.1:35437)
0929 03:00:16.963650 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:16.963657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:16.964190 (+ 533us) server_negotiation.cc:407] Connection header received
0929 03:00:16.964347 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:16.964351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:16.964421 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:16.964502 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:16.966101 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:16.966914 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:16.968205 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:16.968684 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:16.968798 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:16.971121 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:16.971127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:16.971131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:16.971171 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:16.974419 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:16.975046 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:16.975051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:16.975056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:16.975170 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:16.975554 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:16.975559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:16.975562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:16.975927 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:16.976192 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:16.976485 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 03:00:16.976695 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:00:16.980938 3762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dd5b1cd11ef44c0b33ba4933d2d7cdb. Latest consensus state: current_term: 1 leader_uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35437 } } }
I20260929 03:00:16.981369 3762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:16.981803 3761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd5b1cd11ef44c0b33ba4933d2d7cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35437 } } }
I20260929 03:00:16.982061 3761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:16.982304 3769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:16.988147 3769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:16.996941 3769 catalog_manager.cc:1347] Generated new cluster ID: 02b479bf853f433aaefd8ddfd2c1b4a9
I20260929 03:00:16.997066 3769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:17.006696 3769 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:17.008045 3769 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:17.014478 3769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dd5b1cd11ef44c0b33ba4933d2d7cdb: Generated new TSK 0
I20260929 03:00:17.015324 3769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:17.022454 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3683
I20260929 03:00:17.023572 3745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:17.113737 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 3683
2026-09-29T03:00:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1880 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:17 dist-test-slave-q5f0 krb5kdc[3782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:17 dist-test-slave-q5f0 krb5kdc[3782](info): set up 2 sockets
Sep 29 03:00:17 dist-test-slave-q5f0 krb5kdc[3782](info): commencing operation
krb5kdc: starting...
W20260929 03:00:18.132032 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:18 dist-test-slave-q5f0 krb5kdc[3782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:18.379681 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:18.708788 3795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:18.709108 3795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:18.709232 3795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:18.716773 3795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:18.716905 3795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:18.716979 3795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:18.717068 3795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:18.717128 3795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:18.727325 3795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:18.728699 3795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:18.730046 3795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:18.741847 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:18.741874 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:18.741899 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:18.742678 3795 server_base.cc:1034] running on GCE node
I20260929 03:00:18.743438 3795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:18.744961 3795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:18.746201 3795 hybrid_clock.cc:648] HybridClock initialized: now 1790650818746147 us; error 42 us; skew 500 ppm
I20260929 03:00:18.746583 3795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:18 dist-test-slave-q5f0 krb5kdc[3782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:18.753585 3795 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:18.755492 3795 webserver.cc:460] Webserver started at http://127.0.0.1:41975/ using document root <none> and password file <none>
I20260929 03:00:18.756306 3795 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:18.756429 3795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:18.756747 3795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:18.760007 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "67328854cef841ab879329697bd4d7df"
format_stamp: "Formatted at 2026-09-29 03:00:18 on dist-test-slave-q5f0"
server_key: "5ca2d6e8761c82da96cf388b6b64a0e1"
server_key_iv: "42b07ffc1f3e5f80d9b5a521d3f17614"
server_key_version: "encryptionkey@0"
I20260929 03:00:18.760852 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "67328854cef841ab879329697bd4d7df"
format_stamp: "Formatted at 2026-09-29 03:00:18 on dist-test-slave-q5f0"
server_key: "5ca2d6e8761c82da96cf388b6b64a0e1"
server_key_iv: "42b07ffc1f3e5f80d9b5a521d3f17614"
server_key_version: "encryptionkey@0"
I20260929 03:00:18.766929 3795 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 03:00:18.770936 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:18.772569 3795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:00:18.772758 3795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "67328854cef841ab879329697bd4d7df"
format_stamp: "Formatted at 2026-09-29 03:00:18 on dist-test-slave-q5f0"
server_key: "5ca2d6e8761c82da96cf388b6b64a0e1"
server_key_iv: "42b07ffc1f3e5f80d9b5a521d3f17614"
server_key_version: "encryptionkey@0"
I20260929 03:00:18.772964 3795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:18.793779 3795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:18.798377 3795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:18.798662 3795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:18.815621 3795 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46573
I20260929 03:00:18.815621 3861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46573 every 8 connection(s)
I20260929 03:00:18.817037 3795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:18.820853 3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:18.823985 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 3795
I20260929 03:00:18.824151 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:18.824574 15433 external_mini_cluster.cc:1428] Setting key 7688fcc25c36a8f0bce512a1414e8acb
I20260929 03:00:18.832387 3862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df: Bootstrap starting.
I20260929 03:00:18.835608 3862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:18.836617 3862 log.cc:824] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:18.839283 3862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df: No bootstrap required, opened a new log
I20260929 03:00:18.843195 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:18.826441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41257 (local address 127.0.0.1:46573)
0929 03:00:18.827191 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:18.827205 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:18.827247 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:00:18.828965 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:18.828993 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:18.829474 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:18.829860 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:18.830449 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:18.831802 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:18.833171 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:18.833862 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:18.834030 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:18.840412 (+ 6382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:18.840422 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:18.840437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:18.841086 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:18.841195 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:18.841230 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:18.841392 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:18.841665 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 03:00:18.842009 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":313,"threads_started":1}
I20260929 03:00:18.848040 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:18.848214 3862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67328854cef841ab879329697bd4d7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } }
I20260929 03:00:18.848696 3862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:18.848809 3862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67328854cef841ab879329697bd4d7df, State: Initialized, Role: FOLLOWER
I20260929 03:00:18.849509 3862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67328854cef841ab879329697bd4d7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } }
I20260929 03:00:18.849797 3862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:18.849933 3862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:18.850102 3862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:18.853394 3862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67328854cef841ab879329697bd4d7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } }
I20260929 03:00:18.854089 3862 leader_election.cc:302] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67328854cef841ab879329697bd4d7df; no voters:
I20260929 03:00:18.854791 3862 leader_election.cc:288] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:18.854897 3872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:18.856040 3872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [term 1 LEADER]: Becoming Leader. State: Replica: 67328854cef841ab879329697bd4d7df, State: Running, Role: LEADER
I20260929 03:00:18.856699 3872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67328854cef841ab879329697bd4d7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } }
I20260929 03:00:18.857260 3862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:18.859534 3873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67328854cef841ab879329697bd4d7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67328854cef841ab879329697bd4d7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } } }
I20260929 03:00:18.859998 3873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:18.861661 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:18.853356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57975 (local address 127.0.0.1:46573)
0929 03:00:18.853556 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:18.853563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:18.853936 (+ 373us) server_negotiation.cc:407] Connection header received
0929 03:00:18.854117 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:18.854122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:18.854203 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:18.854316 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:18.854860 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:18.855972 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:18.857466 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:18.857916 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:18.858031 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:18.860484 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:18.860491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:18.860497 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:18.860524 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:18.860579 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:18.860600 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:18.860682 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:18.861187 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 03:00:18.861292 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:00:18.861960 3874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67328854cef841ab879329697bd4d7df. Latest consensus state: current_term: 1 leader_uuid: "67328854cef841ab879329697bd4d7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67328854cef841ab879329697bd4d7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } } }
I20260929 03:00:18.862294 3874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:18.864961 3878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:18.870230 3878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:18.879611 3878 catalog_manager.cc:1347] Generated new cluster ID: b80b88608e1f48b1be65057f418ece8d
I20260929 03:00:18.879729 3878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:18.898397 3878 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:18.900377 3878 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:18.913609 3878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67328854cef841ab879329697bd4d7df: Generated new TSK 0
I20260929 03:00:18.914729 3878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:18.922983 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3795
W20260929 03:00:18.924494 3857 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41257
W20260929 03:00:18.925350 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3795: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:18.925511 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 3795
2026-09-29T03:00:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1815 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:19 dist-test-slave-q5f0 krb5kdc[3893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:19 dist-test-slave-q5f0 krb5kdc[3893](info): set up 2 sockets
Sep 29 03:00:19 dist-test-slave-q5f0 krb5kdc[3893](info): commencing operation
krb5kdc: starting...
W20260929 03:00:19.955955 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:20 dist-test-slave-q5f0 krb5kdc[3893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:20.206668 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:20.534914 3906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:20.535223 3906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:20.535315 3906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:20.543051 3906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:20.543197 3906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:20.543244 3906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:20.543303 3906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:20.543346 3906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:20.553524 3906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:20.554827 3906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:20.556157 3906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:20.568091 3911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:20.568126 3912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:20.568063 3914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:20.569190 3906 server_base.cc:1034] running on GCE node
I20260929 03:00:20.570011 3906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:20.571776 3906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:20.573033 3906 hybrid_clock.cc:648] HybridClock initialized: now 1790650820572970 us; error 60 us; skew 500 ppm
I20260929 03:00:20.573424 3906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:20 dist-test-slave-q5f0 krb5kdc[3893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:20.576745 3906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:20.578315 3906 webserver.cc:460] Webserver started at http://127.0.0.1:40377/ using document root <none> and password file <none>
I20260929 03:00:20.578982 3906 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:20.579082 3906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:20.579344 3906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:20.582422 3906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "f4db477ee87d4830a40e24ee8e62a787"
format_stamp: "Formatted at 2026-09-29 03:00:20 on dist-test-slave-q5f0"
server_key: "6ab7c2b9073f4903148d11ca0b6b5cc5"
server_key_iv: "9162c077cc01c61eada9ee95d4d567e1"
server_key_version: "encryptionkey@0"
I20260929 03:00:20.583149 3906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "f4db477ee87d4830a40e24ee8e62a787"
format_stamp: "Formatted at 2026-09-29 03:00:20 on dist-test-slave-q5f0"
server_key: "6ab7c2b9073f4903148d11ca0b6b5cc5"
server_key_iv: "9162c077cc01c61eada9ee95d4d567e1"
server_key_version: "encryptionkey@0"
I20260929 03:00:20.588748 3906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 03:00:20.592520 3921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:20.594033 3906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:00:20.594209 3906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "f4db477ee87d4830a40e24ee8e62a787"
format_stamp: "Formatted at 2026-09-29 03:00:20 on dist-test-slave-q5f0"
server_key: "6ab7c2b9073f4903148d11ca0b6b5cc5"
server_key_iv: "9162c077cc01c61eada9ee95d4d567e1"
server_key_version: "encryptionkey@0"
I20260929 03:00:20.594424 3906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:20.600564 3906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:20.604895 3906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:20.605182 3906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:20.623090 3906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35537
I20260929 03:00:20.623118 3972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35537 every 8 connection(s)
I20260929 03:00:20.624637 3906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:20.628712 3973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:20.631068 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 3906
I20260929 03:00:20.631266 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:20.631933 15433 external_mini_cluster.cc:1428] Setting key 409de8932d1563293ea73be0214176ef
I20260929 03:00:20.640021 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:20.633861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57073 (local address 127.0.0.1:35537)
0929 03:00:20.634697 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:20.634711 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:20.634755 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:00:20.636483 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:20.636507 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:20.636898 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:20.637244 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:20.638352 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:20.638359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:20.638373 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:20.638389 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:20.638457 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:20.638481 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:20.638660 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:20.638760 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 03:00:20.639005 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":389,"threads_started":1}
I20260929 03:00:20.641183 3973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787: Bootstrap starting.
I20260929 03:00:20.643728 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:20.644565 3973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:20.645820 3973 log.cc:824] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:20.648999 3973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787: No bootstrap required, opened a new log
I20260929 03:00:20.651221 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:20.648834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43309 (local address 127.0.0.1:35537)
0929 03:00:20.649016 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:20.649022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:20.649294 (+ 272us) server_negotiation.cc:407] Connection header received
0929 03:00:20.649470 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:20.649475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:20.649549 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:20.649679 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:20.650538 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:20.650544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:20.650548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:20.650567 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:20.650614 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:20.650637 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:20.650755 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:20.650837 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:00:20.650913 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:00:20.657893 3973 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4db477ee87d4830a40e24ee8e62a787" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260929 03:00:20.658268 3973 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:20.658341 3973 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4db477ee87d4830a40e24ee8e62a787, State: Initialized, Role: FOLLOWER
I20260929 03:00:20.658844 3973 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4db477ee87d4830a40e24ee8e62a787" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260929 03:00:20.659058 3973 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:20.659159 3973 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:20.659273 3973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:20.661408 3973 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4db477ee87d4830a40e24ee8e62a787" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260929 03:00:20.661899 3973 leader_election.cc:302] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4db477ee87d4830a40e24ee8e62a787; no voters:
I20260929 03:00:20.662443 3973 leader_election.cc:288] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:20.662588 3983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:20.663576 3983 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [term 1 LEADER]: Becoming Leader. State: Replica: f4db477ee87d4830a40e24ee8e62a787, State: Running, Role: LEADER
I20260929 03:00:20.664132 3983 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4db477ee87d4830a40e24ee8e62a787" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260929 03:00:20.664484 3973 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:20.666536 3984 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4db477ee87d4830a40e24ee8e62a787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4db477ee87d4830a40e24ee8e62a787" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } } }
I20260929 03:00:20.666851 3984 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:20.667655 3990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:20.667500 3985 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4db477ee87d4830a40e24ee8e62a787. Latest consensus state: current_term: 1 leader_uuid: "f4db477ee87d4830a40e24ee8e62a787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4db477ee87d4830a40e24ee8e62a787" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } } }
I20260929 03:00:20.667855 3985 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:20.673214 3990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:20.682581 3990 catalog_manager.cc:1347] Generated new cluster ID: 41efcfba6946442d82cd408dd2e3fe41
I20260929 03:00:20.682722 3990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:20.698925 3990 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:20.699442 3990 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:20.710816 3990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4db477ee87d4830a40e24ee8e62a787: Generated new TSK 0
I20260929 03:00:20.712029 3990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:20.728705 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3906
W20260929 03:00:20.730223 3968 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57073
W20260929 03:00:20.730952 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 3906: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:20.731108 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 3906
2026-09-29T03:00:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1801 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:20 dist-test-slave-q5f0 krb5kdc[4004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:20 dist-test-slave-q5f0 krb5kdc[4004](info): set up 2 sockets
Sep 29 03:00:20 dist-test-slave-q5f0 krb5kdc[4004](info): commencing operation
krb5kdc: starting...
W20260929 03:00:21.747977 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:21 dist-test-slave-q5f0 krb5kdc[4004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:21.990142 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:22.317582 4017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:22.317898 4017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:22.318008 4017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:22.325456 4017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:22.325590 4017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:22.325644 4017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:22.325704 4017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:22.325748 4017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:22.336021 4017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:22.337345 4017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:22.338634 4017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:22.350342 4022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:22.350370 4023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:22.350428 4025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:22.351046 4017 server_base.cc:1034] running on GCE node
I20260929 03:00:22.351838 4017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:22.353277 4017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:22.354491 4017 hybrid_clock.cc:648] HybridClock initialized: now 1790650822354438 us; error 38 us; skew 500 ppm
I20260929 03:00:22.354832 4017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:22 dist-test-slave-q5f0 krb5kdc[4004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:22.358243 4017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:22.359851 4017 webserver.cc:460] Webserver started at http://127.0.0.1:36843/ using document root <none> and password file <none>
I20260929 03:00:22.360549 4017 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:22.360654 4017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:22.360954 4017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:22.364081 4017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "ebc92a8f120442c8ac5f8b159f5db680"
format_stamp: "Formatted at 2026-09-29 03:00:22 on dist-test-slave-q5f0"
server_key: "b3091e659b57cd8a41de9a9eb428622c"
server_key_iv: "346ab643e3040366932ca37967550b4f"
server_key_version: "encryptionkey@0"
I20260929 03:00:22.364816 4017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "ebc92a8f120442c8ac5f8b159f5db680"
format_stamp: "Formatted at 2026-09-29 03:00:22 on dist-test-slave-q5f0"
server_key: "b3091e659b57cd8a41de9a9eb428622c"
server_key_iv: "346ab643e3040366932ca37967550b4f"
server_key_version: "encryptionkey@0"
I20260929 03:00:22.369992 4017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:00:22.373459 4032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:22.374782 4017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:00:22.374966 4017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "ebc92a8f120442c8ac5f8b159f5db680"
format_stamp: "Formatted at 2026-09-29 03:00:22 on dist-test-slave-q5f0"
server_key: "b3091e659b57cd8a41de9a9eb428622c"
server_key_iv: "346ab643e3040366932ca37967550b4f"
server_key_version: "encryptionkey@0"
I20260929 03:00:22.375150 4017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:22.387719 4017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:22.392161 4017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:22.392454 4017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:22.409976 4017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34737
I20260929 03:00:22.409997 4083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34737 every 8 connection(s)
I20260929 03:00:22.411404 4017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:22.414448 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 4017
I20260929 03:00:22.414634 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:22.415163 15433 external_mini_cluster.cc:1428] Setting key 9923344fb17de7a06bf4b0b49e024806
I20260929 03:00:22.415684 4084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 03:00:22 dist-test-slave-q5f0 krb5kdc[4004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:00:22.430511 4084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680: Bootstrap starting.
I20260929 03:00:22.433949 4084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:22.435016 4084 log.cc:824] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:22.438180 4084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680: No bootstrap required, opened a new log
I20260929 03:00:22.439003 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:22.416920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:38695 (local address 127.0.0.1:34737)
0929 03:00:22.417620 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:22.417631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:22.417670 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:00:22.419142 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:22.419163 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:22.419608 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:22.419955 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:22.421802 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:22.424496 (+ 2694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:22.426173 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:22.426885 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:22.427028 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:22.431841 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:22.431849 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:22.431864 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:22.431908 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:22.434842 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:22.435805 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:22.435813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:22.435822 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:22.436033 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:22.436468 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:22.436475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:22.436477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:22.436982 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:22.437219 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:22.437664 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 03:00:22.437932 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":301,"threads_started":1}
I20260929 03:00:22.442647 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:22.447214 4084 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc92a8f120442c8ac5f8b159f5db680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34737 } }
I20260929 03:00:22.447693 4084 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:22.447803 4084 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc92a8f120442c8ac5f8b159f5db680, State: Initialized, Role: FOLLOWER
I20260929 03:00:22.448526 4084 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc92a8f120442c8ac5f8b159f5db680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34737 } }
I20260929 03:00:22.448778 4084 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:22.448936 4084 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:22.449100 4084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:22.451948 4084 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc92a8f120442c8ac5f8b159f5db680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34737 } }
I20260929 03:00:22.452517 4084 leader_election.cc:302] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebc92a8f120442c8ac5f8b159f5db680; no voters:
I20260929 03:00:22.453166 4084 leader_election.cc:288] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:22.453305 4094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:22.454820 4094 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [term 1 LEADER]: Becoming Leader. State: Replica: ebc92a8f120442c8ac5f8b159f5db680, State: Running, Role: LEADER
I20260929 03:00:22.455256 4084 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:22.455404 4094 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc92a8f120442c8ac5f8b159f5db680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34737 } }
I20260929 03:00:22.464794 4095 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebc92a8f120442c8ac5f8b159f5db680" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc92a8f120442c8ac5f8b159f5db680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34737 } } }
I20260929 03:00:22.465219 4095 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:22.465749 4096 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc92a8f120442c8ac5f8b159f5db680. Latest consensus state: current_term: 1 leader_uuid: "ebc92a8f120442c8ac5f8b159f5db680" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc92a8f120442c8ac5f8b159f5db680" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34737 } } }
I20260929 03:00:22.466039 4096 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:22.468475 4108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:22.469023 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:22.447832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:51933 (local address 127.0.0.1:34737)
0929 03:00:22.447997 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:22.448001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:22.448361 (+ 360us) server_negotiation.cc:407] Connection header received
0929 03:00:22.448544 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:22.448548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:22.448610 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:22.448740 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:22.450494 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:22.451303 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:22.452927 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:22.453625 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:22.453760 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:22.457040 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:22.457048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:22.457054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:22.457098 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:22.464769 (+ 7671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:22.465793 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:22.465799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:22.465802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:22.466035 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:22.466638 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:22.466644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:22.466648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:22.467045 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:22.467361 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:22.468481 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0929 03:00:22.468676 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:00:22.473333 4108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:22.482295 4108 catalog_manager.cc:1347] Generated new cluster ID: aaf0bff36f384512a2dd9f9bc07aba19
I20260929 03:00:22.482424 4108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:22.527069 4108 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:22.528627 4108 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:22.536893 4108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebc92a8f120442c8ac5f8b159f5db680: Generated new TSK 0
I20260929 03:00:22.537801 4108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:22.553572 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4017
I20260929 03:00:22.554836 4079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:22.649945 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 4017
W20260929 03:00:22.664001 4002 connection.cc:535] client connection to 127.0.0.1:34737 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34737): Connection reset by peer (error 104)
2026-09-29T03:00:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:22 dist-test-slave-q5f0 krb5kdc[4116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:22 dist-test-slave-q5f0 krb5kdc[4116](info): set up 2 sockets
Sep 29 03:00:22 dist-test-slave-q5f0 krb5kdc[4116](info): commencing operation
krb5kdc: starting...
W20260929 03:00:23.664063 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:23 dist-test-slave-q5f0 krb5kdc[4116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:23.914409 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:24.240578 4129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:24.240936 4129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:24.241073 4129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:24.248783 4129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:24.248924 4129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:24.249014 4129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:24.249100 4129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:24.249166 4129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:24.259552 4129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:24.260908 4129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:24.262297 4129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:24.274492 4135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:24.274497 4134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:24.274482 4137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:24.275254 4129 server_base.cc:1034] running on GCE node
I20260929 03:00:24.276170 4129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:24.277868 4129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:24.279101 4129 hybrid_clock.cc:648] HybridClock initialized: now 1790650824279049 us; error 45 us; skew 500 ppm
I20260929 03:00:24.279480 4129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:24 dist-test-slave-q5f0 krb5kdc[4116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:24.289099 4129 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:24.290771 4129 webserver.cc:460] Webserver started at http://127.0.0.1:38127/ using document root <none> and password file <none>
I20260929 03:00:24.291481 4129 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:24.291654 4129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:24.291962 4129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:24.295159 4129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "708505a7e18a4fca837989fa61d94e19"
format_stamp: "Formatted at 2026-09-29 03:00:24 on dist-test-slave-q5f0"
server_key: "17fb6bc6a9e9f3fde5a5d08529f0e380"
server_key_iv: "67118f2d995ae911582770b5fed136eb"
server_key_version: "encryptionkey@0"
I20260929 03:00:24.296075 4129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "708505a7e18a4fca837989fa61d94e19"
format_stamp: "Formatted at 2026-09-29 03:00:24 on dist-test-slave-q5f0"
server_key: "17fb6bc6a9e9f3fde5a5d08529f0e380"
server_key_iv: "67118f2d995ae911582770b5fed136eb"
server_key_version: "encryptionkey@0"
I20260929 03:00:24.302000 4129 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 03:00:24.305962 4144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:24.307574 4129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:00:24.307762 4129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "708505a7e18a4fca837989fa61d94e19"
format_stamp: "Formatted at 2026-09-29 03:00:24 on dist-test-slave-q5f0"
server_key: "17fb6bc6a9e9f3fde5a5d08529f0e380"
server_key_iv: "67118f2d995ae911582770b5fed136eb"
server_key_version: "encryptionkey@0"
I20260929 03:00:24.307958 4129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:24.337178 4129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:24.341558 4129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:24.341823 4129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:24.359452 4129 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39447
I20260929 03:00:24.359481 4195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39447 every 8 connection(s)
I20260929 03:00:24.360837 4129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:24.364601 4196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:24.368713 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 4129
I20260929 03:00:24.368893 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:24.369462 15433 external_mini_cluster.cc:1428] Setting key 3dd141ec83c3d9d7cf8ffaaf03dac9aa
I20260929 03:00:24.378023 4196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19: Bootstrap starting.
I20260929 03:00:24.381668 4196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:24.382897 4196 log.cc:824] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:24.383061 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:24.371434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:54215 (local address 127.0.0.1:39447)
0929 03:00:24.372210 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:24.372222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:24.372263 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:00:24.373867 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:24.373892 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:24.374373 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:24.374752 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:24.375373 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:24.376753 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:24.378571 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:24.379253 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:24.379406 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:24.380703 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:24.380710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:24.380725 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:24.381308 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:24.381392 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:24.381419 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:24.381533 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:24.381891 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 03:00:24.382062 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":328,"threads_started":1}
I20260929 03:00:24.386020 4196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19: No bootstrap required, opened a new log
I20260929 03:00:24.386996 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:24.395892 4196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708505a7e18a4fca837989fa61d94e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260929 03:00:24.396416 4196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:24.396529 4196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708505a7e18a4fca837989fa61d94e19, State: Initialized, Role: FOLLOWER
I20260929 03:00:24.397195 4196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708505a7e18a4fca837989fa61d94e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260929 03:00:24.397481 4196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:24.397621 4196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:24.397802 4196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:24.398861 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:24.392254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:36371 (local address 127.0.0.1:39447)
0929 03:00:24.392483 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:24.392489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:24.392814 (+ 325us) server_negotiation.cc:407] Connection header received
0929 03:00:24.393016 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:24.393020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:24.393091 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:24.393170 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:24.393770 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:24.394544 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:24.396341 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:24.396874 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:24.397007 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:24.398141 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:24.398146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:24.398149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:24.398167 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:24.398203 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:24.398218 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:24.398294 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:24.398488 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 03:00:24.398538 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:00:24.400688 4196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708505a7e18a4fca837989fa61d94e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260929 03:00:24.401194 4196 leader_election.cc:302] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 708505a7e18a4fca837989fa61d94e19; no voters:
I20260929 03:00:24.401741 4196 leader_election.cc:288] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:24.401835 4206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:24.402500 4206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [term 1 LEADER]: Becoming Leader. State: Replica: 708505a7e18a4fca837989fa61d94e19, State: Running, Role: LEADER
I20260929 03:00:24.403106 4206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708505a7e18a4fca837989fa61d94e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260929 03:00:24.403990 4196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:24.406009 4208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 708505a7e18a4fca837989fa61d94e19. Latest consensus state: current_term: 1 leader_uuid: "708505a7e18a4fca837989fa61d94e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708505a7e18a4fca837989fa61d94e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } } }
I20260929 03:00:24.406417 4208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:24.410209 4207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "708505a7e18a4fca837989fa61d94e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708505a7e18a4fca837989fa61d94e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } } }
I20260929 03:00:24.410429 4207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:24.411695 4213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:24.416162 4213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:24.424232 4213 catalog_manager.cc:1347] Generated new cluster ID: 4526870f7a5b44248294967646366c87
I20260929 03:00:24.424353 4213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:24.436615 4213 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:24.437845 4213 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:24.448210 4213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 708505a7e18a4fca837989fa61d94e19: Generated new TSK 0
I20260929 03:00:24.449025 4213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:24.458244 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4129
W20260929 03:00:24.459561 4191 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.88:54215
W20260929 03:00:24.460300 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4129: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:24.460439 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 4129
W20260929 03:00:24.473815 4114 connection.cc:535] client connection to 127.0.0.1:39447 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39447): Connection reset by peer (error 104)
2026-09-29T03:00:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1809 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:24 dist-test-slave-q5f0 krb5kdc[4227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:24 dist-test-slave-q5f0 krb5kdc[4227](info): set up 2 sockets
Sep 29 03:00:24 dist-test-slave-q5f0 krb5kdc[4227](info): commencing operation
krb5kdc: starting...
W20260929 03:00:25.480124 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.008s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:25 dist-test-slave-q5f0 krb5kdc[4227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:25.728627 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:26.053782 4240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:26.054087 4240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:26.054210 4240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:26.061606 4240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:26.061748 4240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:26.061820 4240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:26.061906 4240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:26.061962 4240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:26.072518 4240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:26.073916 4240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:26.075320 4240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:26.087622 4248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:26.087679 4245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:26.087659 4246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:26.088783 4240 server_base.cc:1034] running on GCE node
I20260929 03:00:26.089603 4240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:26.091040 4240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:26.092247 4240 hybrid_clock.cc:648] HybridClock initialized: now 1790650826092190 us; error 46 us; skew 500 ppm
I20260929 03:00:26.092607 4240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:26 dist-test-slave-q5f0 krb5kdc[4227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:26.095750 4240 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:26.097180 4240 webserver.cc:460] Webserver started at http://127.0.0.1:42059/ using document root <none> and password file <none>
I20260929 03:00:26.097826 4240 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:26.097925 4240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:26.098170 4240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:26.101166 4240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "c414f835a10b440b9059ba3aa44a5801"
format_stamp: "Formatted at 2026-09-29 03:00:26 on dist-test-slave-q5f0"
server_key: "64c74a33fd2777e9927e4d40b0f69db9"
server_key_iv: "22187842e054b495c406cf5302730193"
server_key_version: "encryptionkey@0"
I20260929 03:00:26.101923 4240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "c414f835a10b440b9059ba3aa44a5801"
format_stamp: "Formatted at 2026-09-29 03:00:26 on dist-test-slave-q5f0"
server_key: "64c74a33fd2777e9927e4d40b0f69db9"
server_key_iv: "22187842e054b495c406cf5302730193"
server_key_version: "encryptionkey@0"
I20260929 03:00:26.107230 4240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 03:00:26.111049 4255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:26.112556 4240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:00:26.112708 4240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "c414f835a10b440b9059ba3aa44a5801"
format_stamp: "Formatted at 2026-09-29 03:00:26 on dist-test-slave-q5f0"
server_key: "64c74a33fd2777e9927e4d40b0f69db9"
server_key_iv: "22187842e054b495c406cf5302730193"
server_key_version: "encryptionkey@0"
I20260929 03:00:26.112895 4240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:26.118613 4240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:26.122745 4240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:26.122984 4240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:26.140594 4240 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36603
I20260929 03:00:26.140615 4306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36603 every 8 connection(s)
I20260929 03:00:26.142180 4240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:26.146050 4307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:26.152057 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 4240
I20260929 03:00:26.152238 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:26.152678 15433 external_mini_cluster.cc:1428] Setting key 4eed6019d70d5dc3b854676a9adcb793
I20260929 03:00:26.157061 4307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801: Bootstrap starting.
I20260929 03:00:26.160512 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:26.154501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:33227 (local address 127.0.0.1:36603)
0929 03:00:26.155352 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:26.155364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:26.155407 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:00:26.157114 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:26.157137 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:26.157530 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:26.157848 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:26.158966 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:26.158973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:26.158986 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:26.159002 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:26.159070 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:26.159090 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:26.159258 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:26.159374 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 03:00:26.159584 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":378,"threads_started":1}
I20260929 03:00:26.160597 4307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:26.161914 4307 log.cc:824] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:26.164516 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:26.164990 4307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801: No bootstrap required, opened a new log
I20260929 03:00:26.171420 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:26.168758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:39509 (local address 127.0.0.1:36603)
0929 03:00:26.168950 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:26.168957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:26.169347 (+ 390us) server_negotiation.cc:407] Connection header received
0929 03:00:26.169514 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:26.169520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:26.169589 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:26.169710 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:26.170705 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:26.170711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:26.170716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:26.170735 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:26.170784 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:26.170803 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:26.170938 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:26.171051 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:00:26.171123 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:00:26.173166 4307 raft_consensus.cc:348] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c414f835a10b440b9059ba3aa44a5801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36603 } }
I20260929 03:00:26.173543 4307 raft_consensus.cc:374] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:26.173620 4307 raft_consensus.cc:729] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c414f835a10b440b9059ba3aa44a5801, State: Initialized, Role: FOLLOWER
I20260929 03:00:26.174139 4307 consensus_queue.cc:260] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c414f835a10b440b9059ba3aa44a5801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36603 } }
I20260929 03:00:26.174347 4307 raft_consensus.cc:388] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:26.174451 4307 raft_consensus.cc:482] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:26.174575 4307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:26.176788 4307 raft_consensus.cc:504] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c414f835a10b440b9059ba3aa44a5801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36603 } }
I20260929 03:00:26.177273 4307 leader_election.cc:302] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c414f835a10b440b9059ba3aa44a5801; no voters:
I20260929 03:00:26.177827 4307 leader_election.cc:288] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:26.177996 4317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:26.179097 4317 raft_consensus.cc:686] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [term 1 LEADER]: Becoming Leader. State: Replica: c414f835a10b440b9059ba3aa44a5801, State: Running, Role: LEADER
I20260929 03:00:26.179682 4317 consensus_queue.cc:237] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c414f835a10b440b9059ba3aa44a5801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36603 } }
I20260929 03:00:26.180081 4307 sys_catalog.cc:564] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:26.182454 4319 sys_catalog.cc:455] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c414f835a10b440b9059ba3aa44a5801. Latest consensus state: current_term: 1 leader_uuid: "c414f835a10b440b9059ba3aa44a5801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c414f835a10b440b9059ba3aa44a5801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36603 } } }
I20260929 03:00:26.182843 4319 sys_catalog.cc:458] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:26.183306 4318 sys_catalog.cc:455] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c414f835a10b440b9059ba3aa44a5801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c414f835a10b440b9059ba3aa44a5801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36603 } } }
I20260929 03:00:26.183620 4318 sys_catalog.cc:458] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:26.183784 4326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:26.190852 4326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:26.202481 4326 catalog_manager.cc:1347] Generated new cluster ID: 07dde709e0ea43e283dff13d12062833
I20260929 03:00:26.202632 4326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:26.222112 4326 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:26.222709 4326 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:26.229137 4326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c414f835a10b440b9059ba3aa44a5801: Generated new TSK 0
I20260929 03:00:26.230269 4326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:26.244607 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4240
W20260929 03:00:26.246146 4302 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.88:33227
W20260929 03:00:26.246945 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4240: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:26.247112 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 4240
2026-09-29T03:00:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1786 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:26 dist-test-slave-q5f0 krb5kdc[4338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:26 dist-test-slave-q5f0 krb5kdc[4338](info): set up 2 sockets
Sep 29 03:00:26 dist-test-slave-q5f0 krb5kdc[4338](info): commencing operation
krb5kdc: starting...
W20260929 03:00:27.260036 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:27 dist-test-slave-q5f0 krb5kdc[4338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:27.512413 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:27.836786 4351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:27.837098 4351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:27.837193 4351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:27.844550 4351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:27.844673 4351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:27.844720 4351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:27.844779 4351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:27.844822 4351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:27.854815 4351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:27.856153 4351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:27.857431 4351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:27.869235 4356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:27.869222 4359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:27.869274 4357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:27.870177 4351 server_base.cc:1034] running on GCE node
I20260929 03:00:27.870931 4351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:27.872462 4351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:27.873673 4351 hybrid_clock.cc:648] HybridClock initialized: now 1790650827873622 us; error 43 us; skew 500 ppm
I20260929 03:00:27.874018 4351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:27 dist-test-slave-q5f0 krb5kdc[4338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:27.877190 4351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:27.878731 4351 webserver.cc:460] Webserver started at http://127.0.0.1:45431/ using document root <none> and password file <none>
I20260929 03:00:27.879398 4351 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:27.879547 4351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:27.879856 4351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:27.882951 4351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "ed200b8ad23d42af8053c21361f58e20"
format_stamp: "Formatted at 2026-09-29 03:00:27 on dist-test-slave-q5f0"
server_key: "9408c9d69f4941bb0c5c7725a4595104"
server_key_iv: "9458277713beea869bd7ce049bde6d8b"
server_key_version: "encryptionkey@0"
I20260929 03:00:27.883828 4351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "ed200b8ad23d42af8053c21361f58e20"
format_stamp: "Formatted at 2026-09-29 03:00:27 on dist-test-slave-q5f0"
server_key: "9408c9d69f4941bb0c5c7725a4595104"
server_key_iv: "9458277713beea869bd7ce049bde6d8b"
server_key_version: "encryptionkey@0"
I20260929 03:00:27.889405 4351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:00:27.892961 4366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:27.894335 4351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 03:00:27.894495 4351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "ed200b8ad23d42af8053c21361f58e20"
format_stamp: "Formatted at 2026-09-29 03:00:27 on dist-test-slave-q5f0"
server_key: "9408c9d69f4941bb0c5c7725a4595104"
server_key_iv: "9458277713beea869bd7ce049bde6d8b"
server_key_version: "encryptionkey@0"
I20260929 03:00:27.894681 4351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:27.908722 4351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:27.913214 4351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:27.913496 4351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:27.929816 4351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42331
I20260929 03:00:27.929837 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42331 every 8 connection(s)
I20260929 03:00:27.931133 4351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:27.934947 4418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:27.936595 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 4351
I20260929 03:00:27.936782 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:27.937345 15433 external_mini_cluster.cc:1428] Setting key be22e3fcb5636b9126765d0f8e737b2e
I20260929 03:00:27.949162 4418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20: Bootstrap starting.
Sep 29 03:00:27 dist-test-slave-q5f0 krb5kdc[4338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:00:27.952767 4418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:27.953814 4418 log.cc:824] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:27.957070 4418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20: No bootstrap required, opened a new log
I20260929 03:00:27.960872 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:27.939123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:47649 (local address 127.0.0.1:42331)
0929 03:00:27.939887 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:27.939902 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:27.939946 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:00:27.941501 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:27.941522 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:27.941927 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:27.942258 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:27.943951 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:27.945235 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:27.947025 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:27.947814 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:27.947971 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:27.952705 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:27.952714 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:27.952729 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:27.952772 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:27.956163 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:27.956949 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:27.956956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:27.956964 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:27.957191 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:27.957624 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:27.957629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:27.957632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:27.958337 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:27.958661 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:27.959069 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 03:00:27.959440 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":362,"threads_started":1}
I20260929 03:00:27.965126 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:27.966845 4418 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed200b8ad23d42af8053c21361f58e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42331 } }
I20260929 03:00:27.967276 4418 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:27.967381 4418 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed200b8ad23d42af8053c21361f58e20, State: Initialized, Role: FOLLOWER
I20260929 03:00:27.968075 4418 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed200b8ad23d42af8053c21361f58e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42331 } }
I20260929 03:00:27.968365 4418 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:27.968509 4418 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:27.968683 4418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:27.971582 4418 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed200b8ad23d42af8053c21361f58e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42331 } }
I20260929 03:00:27.972095 4418 leader_election.cc:302] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed200b8ad23d42af8053c21361f58e20; no voters:
I20260929 03:00:27.972692 4418 leader_election.cc:288] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:27.972795 4428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:27.975131 4418 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:27.979772 4428 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [term 1 LEADER]: Becoming Leader. State: Replica: ed200b8ad23d42af8053c21361f58e20, State: Running, Role: LEADER
I20260929 03:00:27.980528 4428 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed200b8ad23d42af8053c21361f58e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42331 } }
I20260929 03:00:27.983662 4429 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed200b8ad23d42af8053c21361f58e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed200b8ad23d42af8053c21361f58e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42331 } } }
I20260929 03:00:27.984064 4429 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:27.984552 4430 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed200b8ad23d42af8053c21361f58e20. Latest consensus state: current_term: 1 leader_uuid: "ed200b8ad23d42af8053c21361f58e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed200b8ad23d42af8053c21361f58e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42331 } } }
I20260929 03:00:27.984838 4430 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:27.985767 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:27.970473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:40687 (local address 127.0.0.1:42331)
0929 03:00:27.970657 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:27.970663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:27.971038 (+ 375us) server_negotiation.cc:407] Connection header received
0929 03:00:27.971179 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:27.971185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:27.971266 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:27.971365 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:27.972915 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:27.974081 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:27.975563 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:27.976023 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:27.976138 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:27.978672 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:27.978677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:27.978680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:27.978715 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:27.981582 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:27.982283 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:27.982289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:27.982292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:27.982461 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:27.983047 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:27.983052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:27.983055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:27.983559 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:27.983850 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:27.985152 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0929 03:00:27.985374 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:00:27.996565 4435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:28.001102 4435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:28.008975 4435 catalog_manager.cc:1347] Generated new cluster ID: 40aac394e29644988ffe3b9668d5c1bd
I20260929 03:00:28.009121 4435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:28.033679 4435 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:28.035187 4435 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:28.041756 4435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed200b8ad23d42af8053c21361f58e20: Generated new TSK 0
I20260929 03:00:28.042577 4435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:28.051126 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4351
I20260929 03:00:28.052297 4413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:28.142918 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 4351
W20260929 03:00:28.156775 4336 connection.cc:535] client connection to 127.0.0.1:42331 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42331): Connection reset by peer (error 104)
2026-09-29T03:00:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:28 dist-test-slave-q5f0 krb5kdc[4451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:28 dist-test-slave-q5f0 krb5kdc[4451](info): set up 2 sockets
Sep 29 03:00:28 dist-test-slave-q5f0 krb5kdc[4451](info): commencing operation
krb5kdc: starting...
W20260929 03:00:29.160048 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:29 dist-test-slave-q5f0 krb5kdc[4451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:29.408102 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:29.732163 4464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:29.732451 4464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:29.732582 4464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:29.739785 4464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:29.739904 4464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:29.739977 4464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:29.740051 4464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:29.740108 4464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:29.750057 4464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41233
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:29.751430 4464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:29.752818 4464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:29.764938 4472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:29.765015 4469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:29.764978 4470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:29.765797 4464 server_base.cc:1034] running on GCE node
I20260929 03:00:29.766620 4464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:29.768174 4464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:29.769419 4464 hybrid_clock.cc:648] HybridClock initialized: now 1790650829769365 us; error 43 us; skew 500 ppm
I20260929 03:00:29.769806 4464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:29 dist-test-slave-q5f0 krb5kdc[4451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:29.779229 4464 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:29.780833 4464 webserver.cc:460] Webserver started at http://127.0.0.1:42215/ using document root <none> and password file <none>
I20260929 03:00:29.781562 4464 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:29.781666 4464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:29.781917 4464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:29.784911 4464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "0a3df650cb074739adfe08925ddc6669"
format_stamp: "Formatted at 2026-09-29 03:00:29 on dist-test-slave-q5f0"
server_key: "07dfd0fc703b05898e62c42dcb974c78"
server_key_iv: "5dd3fb3fb39a5143c7a7dac1b3b46b64"
server_key_version: "encryptionkey@0"
I20260929 03:00:29.785624 4464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "0a3df650cb074739adfe08925ddc6669"
format_stamp: "Formatted at 2026-09-29 03:00:29 on dist-test-slave-q5f0"
server_key: "07dfd0fc703b05898e62c42dcb974c78"
server_key_iv: "5dd3fb3fb39a5143c7a7dac1b3b46b64"
server_key_version: "encryptionkey@0"
I20260929 03:00:29.790771 4464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 03:00:29.794461 4479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:29.795931 4464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:00:29.796094 4464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "0a3df650cb074739adfe08925ddc6669"
format_stamp: "Formatted at 2026-09-29 03:00:29 on dist-test-slave-q5f0"
server_key: "07dfd0fc703b05898e62c42dcb974c78"
server_key_iv: "5dd3fb3fb39a5143c7a7dac1b3b46b64"
server_key_version: "encryptionkey@0"
I20260929 03:00:29.796283 4464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:29.811192 4464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:29.816004 4464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:29.816274 4464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:29.833837 4464 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41233
I20260929 03:00:29.833855 4530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41233 every 8 connection(s)
I20260929 03:00:29.835218 4464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:29.839154 4531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:29.842588 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 4464
I20260929 03:00:29.842759 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:29.843217 15433 external_mini_cluster.cc:1428] Setting key 2df5fad65a112fa3a448ee07e1bd6652
I20260929 03:00:29.851258 4531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669: Bootstrap starting.
I20260929 03:00:29.854473 4531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:29.855568 4531 log.cc:824] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:29.856644 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:29.845250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:49049 (local address 127.0.0.1:41233)
0929 03:00:29.846040 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:29.846053 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:29.846098 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:00:29.847894 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:29.847920 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:29.848400 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:29.848798 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:29.849376 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:29.850671 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:29.852109 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:29.852787 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:29.852964 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:29.854299 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:29.854307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:29.854321 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:29.854920 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:29.854993 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:29.855017 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:29.855156 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:29.855453 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 03:00:29.855642 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":317,"threads_started":1}
I20260929 03:00:29.858356 4531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669: No bootstrap required, opened a new log
I20260929 03:00:29.860246 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:29.867848 4531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3df650cb074739adfe08925ddc6669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } }
I20260929 03:00:29.868322 4531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:29.868440 4531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3df650cb074739adfe08925ddc6669, State: Initialized, Role: FOLLOWER
I20260929 03:00:29.869086 4531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3df650cb074739adfe08925ddc6669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } }
I20260929 03:00:29.869371 4531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:29.869541 4531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:29.869719 4531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:29.872118 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:29.865414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:59325 (local address 127.0.0.1:41233)
0929 03:00:29.865611 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:29.865617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:29.865954 (+ 337us) server_negotiation.cc:407] Connection header received
0929 03:00:29.866141 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:29.866146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:29.866239 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:29.866342 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:29.866925 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:29.867791 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:29.869571 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:29.870083 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:29.870181 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:29.871329 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:29.871334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:29.871337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:29.871375 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:29.871419 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:29.871437 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:29.871565 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:29.871771 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 03:00:29.871815 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 03:00:29.872545 4531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3df650cb074739adfe08925ddc6669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } }
I20260929 03:00:29.873107 4531 leader_election.cc:302] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a3df650cb074739adfe08925ddc6669; no voters:
I20260929 03:00:29.873740 4531 leader_election.cc:288] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:29.873823 4541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:29.874465 4541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [term 1 LEADER]: Becoming Leader. State: Replica: 0a3df650cb074739adfe08925ddc6669, State: Running, Role: LEADER
I20260929 03:00:29.874984 4541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3df650cb074739adfe08925ddc6669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } }
I20260929 03:00:29.875801 4531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:29.878254 4543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a3df650cb074739adfe08925ddc6669. Latest consensus state: current_term: 1 leader_uuid: "0a3df650cb074739adfe08925ddc6669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3df650cb074739adfe08925ddc6669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } } }
I20260929 03:00:29.878711 4543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:29.878225 4542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a3df650cb074739adfe08925ddc6669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a3df650cb074739adfe08925ddc6669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } } }
I20260929 03:00:29.879849 4542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:29.880336 4550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:29.885305 4550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:29.895619 4550 catalog_manager.cc:1347] Generated new cluster ID: 864f2c8b3ae1438a95bea0687452d94f
I20260929 03:00:29.895777 4550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:29.923970 4550 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:29.925350 4550 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:29.937752 4550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a3df650cb074739adfe08925ddc6669: Generated new TSK 0
I20260929 03:00:29.938653 4550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:29.953069 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4464
W20260929 03:00:29.954260 4526 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.88:49049
W20260929 03:00:29.955029 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4464: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:29.955165 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 4464
W20260929 03:00:29.969002 4449 connection.cc:535] client connection to 127.0.0.1:41233 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41233): Connection reset by peer (error 104)
2026-09-29T03:00:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1816 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:30 dist-test-slave-q5f0 krb5kdc[4562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:30 dist-test-slave-q5f0 krb5kdc[4562](info): set up 2 sockets
Sep 29 03:00:30 dist-test-slave-q5f0 krb5kdc[4562](info): commencing operation
krb5kdc: starting...
W20260929 03:00:30.980062 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:31 dist-test-slave-q5f0 krb5kdc[4562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:00:31.225088 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:31.552901 4575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:31.553197 4575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:31.553321 4575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:31.564486 4575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:31.564688 4575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:31.564765 4575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:31.564859 4575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:31.564918 4575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:31.575140 4575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:31.576501 4575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:31.577889 4575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:31.588925 4580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:31.588963 4581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:31.588985 4583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:31.589680 4575 server_base.cc:1034] running on GCE node
I20260929 03:00:31.590523 4575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:31.592011 4575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:31.593226 4575 hybrid_clock.cc:648] HybridClock initialized: now 1790650831593179 us; error 32 us; skew 500 ppm
I20260929 03:00:31.593568 4575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:31 dist-test-slave-q5f0 krb5kdc[4562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:31.596863 4575 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:00:31.598354 4575 webserver.cc:460] Webserver started at http://127.0.0.1:43685/ using document root <none> and password file <none>
I20260929 03:00:31.599035 4575 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:31.599136 4575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:31.599395 4575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:31.602445 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "691d06909a2141d5acd718b0d729be32"
format_stamp: "Formatted at 2026-09-29 03:00:31 on dist-test-slave-q5f0"
server_key: "fc50b819fdaf69d27f50c3c91d956218"
server_key_iv: "650cc6fdf52c175d36e202fcb25bf42f"
server_key_version: "encryptionkey@0"
I20260929 03:00:31.603152 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "691d06909a2141d5acd718b0d729be32"
format_stamp: "Formatted at 2026-09-29 03:00:31 on dist-test-slave-q5f0"
server_key: "fc50b819fdaf69d27f50c3c91d956218"
server_key_iv: "650cc6fdf52c175d36e202fcb25bf42f"
server_key_version: "encryptionkey@0"
I20260929 03:00:31.608495 4575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 03:00:31.612413 4590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:31.613986 4575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:00:31.614161 4575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "691d06909a2141d5acd718b0d729be32"
format_stamp: "Formatted at 2026-09-29 03:00:31 on dist-test-slave-q5f0"
server_key: "fc50b819fdaf69d27f50c3c91d956218"
server_key_iv: "650cc6fdf52c175d36e202fcb25bf42f"
server_key_version: "encryptionkey@0"
I20260929 03:00:31.614387 4575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:31.620545 4575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:31.625030 4575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:31.625326 4575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:31.643152 4575 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44865
I20260929 03:00:31.643189 4641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44865 every 8 connection(s)
I20260929 03:00:31.644603 4575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:31.648245 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:31.648700 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 4575
I20260929 03:00:31.648857 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:31.649300 15433 external_mini_cluster.cc:1428] Setting key d67a9233d78543f8557ae9e337bf4832
I20260929 03:00:31.657476 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:31.651071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:58555 (local address 127.0.0.1:44865)
0929 03:00:31.651937 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:31.651949 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:31.651990 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:00:31.654159 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:31.654185 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:31.654541 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:31.654899 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:31.655919 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:31.655925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:31.655938 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:31.655953 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:31.656013 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:31.656037 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:31.656229 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:31.656362 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 03:00:31.656549 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":409,"threads_started":1}
I20260929 03:00:31.661036 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:31.661167 4642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32: Bootstrap starting.
I20260929 03:00:31.664109 4642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:31.665196 4642 log.cc:824] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:31.667371 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:31.665148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:43449 (local address 127.0.0.1:44865)
0929 03:00:31.665326 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:31.665332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:31.665588 (+ 256us) server_negotiation.cc:407] Connection header received
0929 03:00:31.665763 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:31.665767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:31.665824 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:31.665905 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:31.666795 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:31.666801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:31.666805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:31.666822 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:31.666866 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:31.666885 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:31.666995 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:31.667039 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 03:00:31.667096 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:00:31.668298 4642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32: No bootstrap required, opened a new log
I20260929 03:00:31.675935 4642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d06909a2141d5acd718b0d729be32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44865 } }
I20260929 03:00:31.676298 4642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:31.676376 4642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691d06909a2141d5acd718b0d729be32, State: Initialized, Role: FOLLOWER
I20260929 03:00:31.676895 4642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d06909a2141d5acd718b0d729be32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44865 } }
I20260929 03:00:31.677098 4642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:31.677197 4642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:31.677314 4642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:31.679579 4642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d06909a2141d5acd718b0d729be32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44865 } }
I20260929 03:00:31.680096 4642 leader_election.cc:302] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 691d06909a2141d5acd718b0d729be32; no voters:
I20260929 03:00:31.680703 4642 leader_election.cc:288] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:31.680841 4652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:31.681514 4652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [term 1 LEADER]: Becoming Leader. State: Replica: 691d06909a2141d5acd718b0d729be32, State: Running, Role: LEADER
I20260929 03:00:31.682183 4652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d06909a2141d5acd718b0d729be32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44865 } }
I20260929 03:00:31.682890 4642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:31.685274 4653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691d06909a2141d5acd718b0d729be32. Latest consensus state: current_term: 1 leader_uuid: "691d06909a2141d5acd718b0d729be32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d06909a2141d5acd718b0d729be32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44865 } } }
I20260929 03:00:31.685526 4654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "691d06909a2141d5acd718b0d729be32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691d06909a2141d5acd718b0d729be32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44865 } } }
I20260929 03:00:31.685812 4653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:31.685865 4654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:31.686784 4661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:31.692142 4661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:31.701323 4661 catalog_manager.cc:1347] Generated new cluster ID: 95fd23eb3de84b908913488ef43a2008
I20260929 03:00:31.701435 4661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:31.710306 4661 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:31.710852 4661 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:31.721827 4661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 691d06909a2141d5acd718b0d729be32: Generated new TSK 0
I20260929 03:00:31.722672 4661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:31.736449 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4575
W20260929 03:00:31.737709 4637 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.88:58555
W20260929 03:00:31.738422 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4575: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:31.738577 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 4575
2026-09-29T03:00:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1779 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:31 dist-test-slave-q5f0 krb5kdc[4673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:31 dist-test-slave-q5f0 krb5kdc[4673](info): set up 2 sockets
Sep 29 03:00:31 dist-test-slave-q5f0 krb5kdc[4673](info): commencing operation
krb5kdc: starting...
W20260929 03:00:32.752091 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:32 dist-test-slave-q5f0 krb5kdc[4673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 03:00:33.000005 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:33253
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:33253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:33.326009 4686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:33.326359 4686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:33.326491 4686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:33.333920 4686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:33.334041 4686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:33.334095 4686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:33.334178 4686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:33.334231 4686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:33.344321 4686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:33253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:33253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:33.345675 4686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:33.347096 4686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:33.359081 4691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:33.359158 4692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:33.359120 4694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:33.360111 4686 server_base.cc:1034] running on GCE node
I20260929 03:00:33.360957 4686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:33.362532 4686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:33.363767 4686 hybrid_clock.cc:648] HybridClock initialized: now 1790650833363706 us; error 62 us; skew 500 ppm
I20260929 03:00:33.364166 4686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:33 dist-test-slave-q5f0 krb5kdc[4673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650833, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:33.367694 4686 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 03:00:33.369342 4686 webserver.cc:460] Webserver started at http://127.15.18.126:45065/ using document root <none> and password file <none>
I20260929 03:00:33.370095 4686 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:33.370229 4686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:33.370522 4686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:33.373577 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "c27eb6ba52954865bd46527b9f247871"
format_stamp: "Formatted at 2026-09-29 03:00:33 on dist-test-slave-q5f0"
server_key: "31891de470bccf392f13a203fd4fcc0a"
server_key_iv: "ccb17c2b326479f61305ebb099617dc2"
server_key_version: "encryptionkey@0"
I20260929 03:00:33.374317 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "c27eb6ba52954865bd46527b9f247871"
format_stamp: "Formatted at 2026-09-29 03:00:33 on dist-test-slave-q5f0"
server_key: "31891de470bccf392f13a203fd4fcc0a"
server_key_iv: "ccb17c2b326479f61305ebb099617dc2"
server_key_version: "encryptionkey@0"
I20260929 03:00:33.379454 4686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 03:00:33.383010 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:33.384457 4686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:00:33.384624 4686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "c27eb6ba52954865bd46527b9f247871"
format_stamp: "Formatted at 2026-09-29 03:00:33 on dist-test-slave-q5f0"
server_key: "31891de470bccf392f13a203fd4fcc0a"
server_key_iv: "ccb17c2b326479f61305ebb099617dc2"
server_key_version: "encryptionkey@0"
I20260929 03:00:33.384846 4686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:33.396889 4686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:33.401598 4686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:33.401908 4686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:33.420334 4686 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:33253
I20260929 03:00:33.420384 4752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:33253 every 8 connection(s)
I20260929 03:00:33.421752 4686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:33.424199 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 4686
I20260929 03:00:33.424391 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:33.424973 15433 external_mini_cluster.cc:1428] Setting key 1ba337ce5a96e51305398829d765e620
I20260929 03:00:33.426381 4753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 03:00:33 dist-test-slave-q5f0 krb5kdc[4673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 03:00:33.442905 4753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871: Bootstrap starting.
I20260929 03:00:33.446764 4753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:33.447958 4753 log.cc:824] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:33.451061 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:33.428064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51181 (local address 127.15.18.126:33253)
0929 03:00:33.428923 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:33.428941 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:33.428998 (+ 57us) server_negotiation.cc:407] Connection header received
0929 03:00:33.430899 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:33.430930 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:33.431575 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:33.432099 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:33.434001 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:33.436250 (+ 2249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:33.438015 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:33.438723 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:33.438909 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:33.443469 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:33.443479 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:33.443497 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:33.443576 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:33.446658 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:33.447604 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:33.447612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:33.447619 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:33.447803 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:33.448319 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:33.448323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:33.448325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:33.448888 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:33.449235 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:33.449734 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 03:00:33.450049 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":330,"threads_started":1}
I20260929 03:00:33.451171 4753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871: No bootstrap required, opened a new log
I20260929 03:00:33.454926 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:33.460292 4753 raft_consensus.cc:348] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eb6ba52954865bd46527b9f247871" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33253 } }
I20260929 03:00:33.460832 4753 raft_consensus.cc:374] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:33.460959 4753 raft_consensus.cc:729] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c27eb6ba52954865bd46527b9f247871, State: Initialized, Role: FOLLOWER
I20260929 03:00:33.461582 4753 consensus_queue.cc:260] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eb6ba52954865bd46527b9f247871" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33253 } }
I20260929 03:00:33.461809 4753 raft_consensus.cc:388] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:33.461954 4753 raft_consensus.cc:482] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:33.462112 4753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:33.465075 4753 raft_consensus.cc:504] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eb6ba52954865bd46527b9f247871" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33253 } }
I20260929 03:00:33.465672 4753 leader_election.cc:302] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c27eb6ba52954865bd46527b9f247871; no voters:
I20260929 03:00:33.466404 4753 leader_election.cc:288] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:33.466534 4763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:33.467301 4763 raft_consensus.cc:686] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [term 1 LEADER]: Becoming Leader. State: Replica: c27eb6ba52954865bd46527b9f247871, State: Running, Role: LEADER
I20260929 03:00:33.467949 4763 consensus_queue.cc:237] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eb6ba52954865bd46527b9f247871" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33253 } }
I20260929 03:00:33.468652 4753 sys_catalog.cc:564] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:33.470038 4764 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27eb6ba52954865bd46527b9f247871. Latest consensus state: current_term: 1 leader_uuid: "c27eb6ba52954865bd46527b9f247871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eb6ba52954865bd46527b9f247871" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33253 } } }
I20260929 03:00:33.470427 4764 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:33.474987 4768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:33.475679 4765 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c27eb6ba52954865bd46527b9f247871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27eb6ba52954865bd46527b9f247871" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 33253 } } }
I20260929 03:00:33.476022 4765 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:33.478085 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:33.459387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58215 (local address 127.15.18.126:33253)
0929 03:00:33.459606 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:33.459612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:33.460372 (+ 760us) server_negotiation.cc:407] Connection header received
0929 03:00:33.460570 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:33.460575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:33.460659 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:33.460837 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:33.462793 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:33.464035 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:33.465297 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:33.466021 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:33.466173 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:33.469275 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:33.469282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:33.469287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:33.469330 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:33.473701 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:33.474695 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:33.474701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:33.474705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:33.474892 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:33.475590 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:33.475596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:33.475599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:33.476207 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:33.476663 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:33.477415 (+ 752us) server_negotiation.cc:299] Negotiation successful
0929 03:00:33.477626 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:00:33.481164 4768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:33.491429 4768 catalog_manager.cc:1347] Generated new cluster ID: 213c79b298e340ccbabeed940040150b
I20260929 03:00:33.491575 4768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:33.506309 4768 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:33.507913 4768 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:33.513867 4768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c27eb6ba52954865bd46527b9f247871: Generated new TSK 0
I20260929 03:00:33.514780 4768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:33.528769 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4686
I20260929 03:00:33.530769 4748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:33.621315 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 4686
W20260929 03:00:33.634991 4671 connection.cc:535] client connection to 127.15.18.126:33253 recv error: Network error: failed to read from TLS socket (remote: 127.15.18.126:33253): Connection reset by peer (error 104)
2026-09-29T03:00:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:33 dist-test-slave-q5f0 krb5kdc[4785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:33 dist-test-slave-q5f0 krb5kdc[4785](info): set up 2 sockets
Sep 29 03:00:33 dist-test-slave-q5f0 krb5kdc[4785](info): commencing operation
krb5kdc: starting...
W20260929 03:00:34.636165 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:34 dist-test-slave-q5f0 krb5kdc[4785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 03:00:34.887081 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:41189
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:41189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:35.219054 4798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:35.219347 4798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:35.219491 4798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:35.226874 4798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:35.227000 4798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:35.227061 4798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:35.227144 4798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:35.227200 4798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:35.237648 4798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:41189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:41189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:35.238977 4798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:35.240355 4798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:35.252113 4803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:35.252168 4806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:35.252084 4804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:35.253041 4798 server_base.cc:1034] running on GCE node
I20260929 03:00:35.253866 4798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:35.255491 4798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:35.256762 4798 hybrid_clock.cc:648] HybridClock initialized: now 1790650835256689 us; error 68 us; skew 500 ppm
I20260929 03:00:35.257148 4798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:35 dist-test-slave-q5f0 krb5kdc[4785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650835, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:35.264014 4798 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 03:00:35.265678 4798 webserver.cc:460] Webserver started at http://127.15.18.126:42371/ using document root <none> and password file <none>
I20260929 03:00:35.266387 4798 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:35.266508 4798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:35.266793 4798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:35.269946 4798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "6979bda5684f498b9cc769d0914f5b0e"
format_stamp: "Formatted at 2026-09-29 03:00:35 on dist-test-slave-q5f0"
server_key: "baa272bf0931fcceeccfc7c2b7405c16"
server_key_iv: "76b6e3ba55798ca9c26627deff60f609"
server_key_version: "encryptionkey@0"
I20260929 03:00:35.270700 4798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "6979bda5684f498b9cc769d0914f5b0e"
format_stamp: "Formatted at 2026-09-29 03:00:35 on dist-test-slave-q5f0"
server_key: "baa272bf0931fcceeccfc7c2b7405c16"
server_key_iv: "76b6e3ba55798ca9c26627deff60f609"
server_key_version: "encryptionkey@0"
I20260929 03:00:35.276165 4798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 03:00:35.279788 4813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:35.281198 4798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:00:35.281369 4798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "6979bda5684f498b9cc769d0914f5b0e"
format_stamp: "Formatted at 2026-09-29 03:00:35 on dist-test-slave-q5f0"
server_key: "baa272bf0931fcceeccfc7c2b7405c16"
server_key_iv: "76b6e3ba55798ca9c26627deff60f609"
server_key_version: "encryptionkey@0"
I20260929 03:00:35.281555 4798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:35.297183 4798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:35.301507 4798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:35.301777 4798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:35.319332 4798 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:41189
I20260929 03:00:35.319352 4864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:41189 every 8 connection(s)
I20260929 03:00:35.320689 4798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:35.322728 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 4798
I20260929 03:00:35.322911 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:35.323458 15433 external_mini_cluster.cc:1428] Setting key 90885895231bd6e4c6e5ede89d6a763c
I20260929 03:00:35.325286 4865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:35.337709 4868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:35.325323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47279 (local address 127.15.18.126:41189)
0929 03:00:35.326164 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:35.326176 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:35.326225 (+ 49us) server_negotiation.cc:407] Connection header received
0929 03:00:35.328011 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:35.328035 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:35.328502 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:35.328896 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:35.329497 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:35.331024 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:35.332924 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:35.333665 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:35.333860 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:35.335175 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:35.335183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:35.335199 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:35.335909 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:35.335996 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:35.336030 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:35.336164 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:35.336438 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 03:00:35.336636 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":382,"threads_started":1}
I20260929 03:00:35.338408 4865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e: Bootstrap starting.
I20260929 03:00:35.341905 4865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:35.341915 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:35.342913 4865 log.cc:824] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:35.345968 4865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e: No bootstrap required, opened a new log
I20260929 03:00:35.353533 4868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:35.346395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35669 (local address 127.15.18.126:41189)
0929 03:00:35.346603 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:35.346610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:35.347007 (+ 397us) server_negotiation.cc:407] Connection header received
0929 03:00:35.347167 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:35.347170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:35.347230 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:35.347318 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:35.347990 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:35.348802 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:35.350808 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:35.351321 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:35.351445 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:35.352729 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:35.352734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:35.352737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:35.352756 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:35.352848 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:35.352865 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:35.352997 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:35.353166 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 03:00:35.353214 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:00:35.355865 4865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6979bda5684f498b9cc769d0914f5b0e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41189 } }
I20260929 03:00:35.356267 4865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:35.356360 4865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6979bda5684f498b9cc769d0914f5b0e, State: Initialized, Role: FOLLOWER
I20260929 03:00:35.356914 4865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6979bda5684f498b9cc769d0914f5b0e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41189 } }
I20260929 03:00:35.357138 4865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:35.357267 4865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:35.357424 4865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:35.359704 4865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6979bda5684f498b9cc769d0914f5b0e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41189 } }
I20260929 03:00:35.360209 4865 leader_election.cc:302] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6979bda5684f498b9cc769d0914f5b0e; no voters:
I20260929 03:00:35.360800 4865 leader_election.cc:288] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:35.360976 4875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:35.362144 4875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [term 1 LEADER]: Becoming Leader. State: Replica: 6979bda5684f498b9cc769d0914f5b0e, State: Running, Role: LEADER
I20260929 03:00:35.362694 4875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6979bda5684f498b9cc769d0914f5b0e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41189 } }
I20260929 03:00:35.362967 4865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:35.365553 4877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6979bda5684f498b9cc769d0914f5b0e. Latest consensus state: current_term: 1 leader_uuid: "6979bda5684f498b9cc769d0914f5b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6979bda5684f498b9cc769d0914f5b0e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41189 } } }
I20260929 03:00:35.365644 4876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6979bda5684f498b9cc769d0914f5b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6979bda5684f498b9cc769d0914f5b0e" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 41189 } } }
I20260929 03:00:35.365911 4877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:35.366022 4876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:35.366971 4884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:35.374505 4884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:35.385442 4884 catalog_manager.cc:1347] Generated new cluster ID: 7386967b1eeb4377afed04f056ea61a2
I20260929 03:00:35.385578 4884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:35.408025 4884 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:35.410003 4884 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:35.418648 4884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6979bda5684f498b9cc769d0914f5b0e: Generated new TSK 0
I20260929 03:00:35.419881 4884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:35.436902 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4798
W20260929 03:00:35.438159 4860 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47279
W20260929 03:00:35.438911 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4798: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:35.439071 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 4798
W20260929 03:00:35.452983 4783 connection.cc:535] client connection to 127.15.18.126:41189 recv error: Network error: failed to read from TLS socket (remote: 127.15.18.126:41189): Connection reset by peer (error 104)
2026-09-29T03:00:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1818 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:35 dist-test-slave-q5f0 krb5kdc[4896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:35 dist-test-slave-q5f0 krb5kdc[4896](info): set up 2 sockets
Sep 29 03:00:35 dist-test-slave-q5f0 krb5kdc[4896](info): commencing operation
krb5kdc: starting...
W20260929 03:00:36.464134 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:36 dist-test-slave-q5f0 krb5kdc[4896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 03:00:36.715147 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:40097
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:40097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:37.040447 4909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:37.040724 4909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:37.040874 4909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:37.048312 4909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:37.048424 4909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:37.048477 4909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:37.048537 4909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:37.048579 4909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:37.058743 4909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:40097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:40097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:37.060076 4909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:37.061375 4909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:37.073122 4914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:37.073140 4917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:37.073167 4915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:37.073980 4909 server_base.cc:1034] running on GCE node
I20260929 03:00:37.074800 4909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:37.076311 4909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:37.077548 4909 hybrid_clock.cc:648] HybridClock initialized: now 1790650837077506 us; error 71 us; skew 500 ppm
I20260929 03:00:37.077927 4909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:37 dist-test-slave-q5f0 krb5kdc[4896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650837, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:37.081082 4909 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 03:00:37.082595 4909 webserver.cc:460] Webserver started at http://127.15.18.126:44443/ using document root <none> and password file <none>
I20260929 03:00:37.083276 4909 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:37.083393 4909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:37.083730 4909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:37.086747 4909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "5f06411996e540f0905b1208710dc977"
format_stamp: "Formatted at 2026-09-29 03:00:37 on dist-test-slave-q5f0"
server_key: "3673d7a35ec2888777854c5ed17f80c8"
server_key_iv: "b1ea22a02c21e080089b8f45c6b17683"
server_key_version: "encryptionkey@0"
I20260929 03:00:37.087546 4909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "5f06411996e540f0905b1208710dc977"
format_stamp: "Formatted at 2026-09-29 03:00:37 on dist-test-slave-q5f0"
server_key: "3673d7a35ec2888777854c5ed17f80c8"
server_key_iv: "b1ea22a02c21e080089b8f45c6b17683"
server_key_version: "encryptionkey@0"
I20260929 03:00:37.093000 4909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 03:00:37.096714 4924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:37.098174 4909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:00:37.098335 4909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "5f06411996e540f0905b1208710dc977"
format_stamp: "Formatted at 2026-09-29 03:00:37 on dist-test-slave-q5f0"
server_key: "3673d7a35ec2888777854c5ed17f80c8"
server_key_iv: "b1ea22a02c21e080089b8f45c6b17683"
server_key_version: "encryptionkey@0"
I20260929 03:00:37.098526 4909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:37.104414 4909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:37.108740 4909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:37.109025 4909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:37.127018 4909 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:40097
I20260929 03:00:37.127036 4975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:40097 every 8 connection(s)
I20260929 03:00:37.128491 4909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:37.129490 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 4909
I20260929 03:00:37.129678 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:37.130265 15433 external_mini_cluster.cc:1428] Setting key 1c59fd8974e8a2ad5daf6674fb55aae2
I20260929 03:00:37.133143 4976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:37.138206 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:37.132243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52925 (local address 127.15.18.126:40097)
0929 03:00:37.133019 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:37.133031 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:37.133073 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:00:37.134726 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:37.134750 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:37.135105 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:37.135473 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:37.136580 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:37.136587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:37.136600 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:37.136621 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:37.136692 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:37.136716 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:37.136885 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:37.136994 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:00:37.137169 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":340,"threads_started":1}
I20260929 03:00:37.142165 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:37.146457 4976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977: Bootstrap starting.
I20260929 03:00:37.149982 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:37.147419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54965 (local address 127.15.18.126:40097)
0929 03:00:37.147634 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:37.147640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:37.148105 (+ 465us) server_negotiation.cc:407] Connection header received
0929 03:00:37.148250 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:37.148255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:37.148321 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:37.148447 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:37.149283 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:37.149288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:37.149292 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:37.149311 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:37.149358 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:37.149380 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:37.149513 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:37.149634 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 03:00:37.149693 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:00:37.150251 4976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:37.151335 4976 log.cc:824] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:37.154263 4976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977: No bootstrap required, opened a new log
I20260929 03:00:37.162459 4976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f06411996e540f0905b1208710dc977" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40097 } }
I20260929 03:00:37.162897 4976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:37.163005 4976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f06411996e540f0905b1208710dc977, State: Initialized, Role: FOLLOWER
I20260929 03:00:37.163770 4976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f06411996e540f0905b1208710dc977" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40097 } }
I20260929 03:00:37.164036 4976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:37.164208 4976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:37.164399 4976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:37.166750 4976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f06411996e540f0905b1208710dc977" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40097 } }
I20260929 03:00:37.167303 4976 leader_election.cc:302] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f06411996e540f0905b1208710dc977; no voters:
I20260929 03:00:37.167939 4976 leader_election.cc:288] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:37.168078 4986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:37.168735 4986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [term 1 LEADER]: Becoming Leader. State: Replica: 5f06411996e540f0905b1208710dc977, State: Running, Role: LEADER
I20260929 03:00:37.169269 4986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f06411996e540f0905b1208710dc977" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40097 } }
I20260929 03:00:37.170076 4976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:37.171886 4987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f06411996e540f0905b1208710dc977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f06411996e540f0905b1208710dc977" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40097 } } }
I20260929 03:00:37.172281 4987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:37.172261 4988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f06411996e540f0905b1208710dc977. Latest consensus state: current_term: 1 leader_uuid: "5f06411996e540f0905b1208710dc977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f06411996e540f0905b1208710dc977" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 40097 } } }
I20260929 03:00:37.172648 4988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:37.175668 4992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:37.180181 4992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:37.188838 4992 catalog_manager.cc:1347] Generated new cluster ID: c0be1d93c854499ca0d19ad48d17500f
I20260929 03:00:37.188962 4992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:37.204625 4992 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:37.205211 4992 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:37.216068 4992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f06411996e540f0905b1208710dc977: Generated new TSK 0
I20260929 03:00:37.216981 4992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:37.228626 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4909
W20260929 03:00:37.229936 4971 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52925
W20260929 03:00:37.230711 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 4909: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:37.230894 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 4909
2026-09-29T03:00:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1791 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:37 dist-test-slave-q5f0 krb5kdc[5007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:37 dist-test-slave-q5f0 krb5kdc[5007](info): set up 2 sockets
Sep 29 03:00:37 dist-test-slave-q5f0 krb5kdc[5007](info): commencing operation
krb5kdc: starting...
W20260929 03:00:38.248066 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:38 dist-test-slave-q5f0 krb5kdc[5007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 03:00:38.491369 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:39635
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:39635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:38.817098 5020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:38.817410 5020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:38.817548 5020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:38.824769 5020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:38.824894 5020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:38.824972 5020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:38.825052 5020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:38.825110 5020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:38.835718 5020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:39635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:39635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:38.837054 5020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:38.838403 5020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:38.849728 5025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:38.849762 5026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:38.849797 5028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:38.850428 5020 server_base.cc:1034] running on GCE node
I20260929 03:00:38.851235 5020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:38.852726 5020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:38.853966 5020 hybrid_clock.cc:648] HybridClock initialized: now 1790650838853911 us; error 42 us; skew 500 ppm
I20260929 03:00:38.854318 5020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:38 dist-test-slave-q5f0 krb5kdc[5007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650838, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:38.857290 5020 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 03:00:38.858726 5020 webserver.cc:460] Webserver started at http://127.15.18.126:39481/ using document root <none> and password file <none>
I20260929 03:00:38.859380 5020 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:38.859494 5020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:38.859802 5020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:38.862711 5020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "595c7729c84046da9829948b1ab3dc4f"
format_stamp: "Formatted at 2026-09-29 03:00:38 on dist-test-slave-q5f0"
server_key: "fb78a19e17a6595ffd760079d3b85a0f"
server_key_iv: "8b93cf950547556d8122803f34d4547d"
server_key_version: "encryptionkey@0"
I20260929 03:00:38.863447 5020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "595c7729c84046da9829948b1ab3dc4f"
format_stamp: "Formatted at 2026-09-29 03:00:38 on dist-test-slave-q5f0"
server_key: "fb78a19e17a6595ffd760079d3b85a0f"
server_key_iv: "8b93cf950547556d8122803f34d4547d"
server_key_version: "encryptionkey@0"
I20260929 03:00:38.868517 5020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 03:00:38.871951 5035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:38.873234 5020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 03:00:38.873399 5020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "595c7729c84046da9829948b1ab3dc4f"
format_stamp: "Formatted at 2026-09-29 03:00:38 on dist-test-slave-q5f0"
server_key: "fb78a19e17a6595ffd760079d3b85a0f"
server_key_iv: "8b93cf950547556d8122803f34d4547d"
server_key_version: "encryptionkey@0"
I20260929 03:00:38.873574 5020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:38.888955 5020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:38.893124 5020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:38.893383 5020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:38.911156 5020 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:39635
I20260929 03:00:38.911180 5086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:39635 every 8 connection(s)
I20260929 03:00:38.912562 5020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:38.915704 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 5020
I20260929 03:00:38.915901 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:38.916322 15433 external_mini_cluster.cc:1428] Setting key d1528bb43d8c7375d75c2a53f9927025
I20260929 03:00:38.916333 5087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:38.930315 5087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f: Bootstrap starting.
Sep 29 03:00:38 dist-test-slave-q5f0 krb5kdc[5007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 03:00:38.933490 5087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:38.934552 5087 log.cc:824] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:38.937310 5087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f: No bootstrap required, opened a new log
I20260929 03:00:38.938758 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:38.917966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59049 (local address 127.15.18.126:39635)
0929 03:00:38.918708 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:38.918719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:38.918759 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:00:38.920767 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:38.920790 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:38.921202 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:38.921543 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:38.923260 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:38.924569 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:38.926280 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:38.926995 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:38.927125 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:38.931659 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:38.931667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:38.931684 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:38.931729 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:38.934700 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:38.935360 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:38.935374 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:38.935382 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:38.935605 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:38.935964 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:38.935969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:38.935973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:38.936505 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:38.936778 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:38.937221 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 03:00:38.937776 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":330,"threads_started":1}
I20260929 03:00:38.942283 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:38.945080 5087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595c7729c84046da9829948b1ab3dc4f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39635 } }
I20260929 03:00:38.945448 5087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:38.945546 5087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595c7729c84046da9829948b1ab3dc4f, State: Initialized, Role: FOLLOWER
I20260929 03:00:38.946089 5087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595c7729c84046da9829948b1ab3dc4f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39635 } }
I20260929 03:00:38.946316 5087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:38.946467 5087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:38.946676 5087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:38.949753 5087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595c7729c84046da9829948b1ab3dc4f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39635 } }
I20260929 03:00:38.950299 5087 leader_election.cc:302] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 595c7729c84046da9829948b1ab3dc4f; no voters:
I20260929 03:00:38.950947 5087 leader_election.cc:288] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:38.951045 5097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:38.953217 5087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:38.957944 5097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [term 1 LEADER]: Becoming Leader. State: Replica: 595c7729c84046da9829948b1ab3dc4f, State: Running, Role: LEADER
I20260929 03:00:38.958717 5097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595c7729c84046da9829948b1ab3dc4f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39635 } }
I20260929 03:00:38.961768 5099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 595c7729c84046da9829948b1ab3dc4f. Latest consensus state: current_term: 1 leader_uuid: "595c7729c84046da9829948b1ab3dc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595c7729c84046da9829948b1ab3dc4f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39635 } } }
I20260929 03:00:38.961745 5098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "595c7729c84046da9829948b1ab3dc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595c7729c84046da9829948b1ab3dc4f" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 39635 } } }
I20260929 03:00:38.962469 5098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:38.963171 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:38.946469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37753 (local address 127.15.18.126:39635)
0929 03:00:38.946663 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:38.946671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:38.947140 (+ 469us) server_negotiation.cc:407] Connection header received
0929 03:00:38.947334 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:38.947337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:38.947413 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:38.947490 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:38.949243 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:38.949975 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:38.951745 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:38.952400 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:38.952494 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:38.955685 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:38.955696 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:38.955700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:38.955739 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:38.959235 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:38.959959 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:38.959963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:38.959965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:38.960099 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:38.960597 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:38.960601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:38.960604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:38.961209 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:38.962410 (+ 1201us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:38.962560 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 03:00:38.962825 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:00:38.962388 5099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f [sys.catalog]: This master's current role is: LEADER
W20260929 03:00:38.967870 5111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:00:38.967998 5111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:00:38.968155 5112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:38.970288 5112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:38.979223 5112 catalog_manager.cc:1347] Generated new cluster ID: c54ea838d1bf46f499304beacc670421
I20260929 03:00:38.979327 5112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:38.992615 5112 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:38.993963 5112 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:39.009114 5112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 595c7729c84046da9829948b1ab3dc4f: Generated new TSK 0
I20260929 03:00:39.010113 5112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:39.017669 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5020
I20260929 03:00:39.019459 5082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:39.124781 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 5020
2026-09-29T03:00:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:39 dist-test-slave-q5f0 krb5kdc[5119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:39 dist-test-slave-q5f0 krb5kdc[5119](info): set up 2 sockets
Sep 29 03:00:39 dist-test-slave-q5f0 krb5kdc[5119](info): commencing operation
krb5kdc: starting...
W20260929 03:00:40.148193 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:40 dist-test-slave-q5f0 krb5kdc[5119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 03:00:40.404918 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:37293
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:44753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:37293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:40.741672 5132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:40.741997 5132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:40.742131 5132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:40.749517 5132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:40.749653 5132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:40.749727 5132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:40.749809 5132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:40.749871 5132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:40.760653 5132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:44753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:37293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:37293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:40.761966 5132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:40.763280 5132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:40.774722 5137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:40.774791 5138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:40.774945 5140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:40.775612 5132 server_base.cc:1034] running on GCE node
I20260929 03:00:40.776386 5132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:40.777854 5132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:40.779096 5132 hybrid_clock.cc:648] HybridClock initialized: now 1790650840779051 us; error 38 us; skew 500 ppm
I20260929 03:00:40.779430 5132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:40 dist-test-slave-q5f0 krb5kdc[5119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650840, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:40.787241 5132 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 03:00:40.788925 5132 webserver.cc:460] Webserver started at http://127.15.18.126:38629/ using document root <none> and password file <none>
I20260929 03:00:40.789623 5132 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:40.789726 5132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:40.789987 5132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:40.793052 5132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "974a0d534419441abc76cb040d722850"
format_stamp: "Formatted at 2026-09-29 03:00:40 on dist-test-slave-q5f0"
server_key: "7e188fb26e44a0d0fce06a89d7fea484"
server_key_iv: "85ecd6151e1e8ade0b4e431ae771eca9"
server_key_version: "encryptionkey@0"
I20260929 03:00:40.793768 5132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "974a0d534419441abc76cb040d722850"
format_stamp: "Formatted at 2026-09-29 03:00:40 on dist-test-slave-q5f0"
server_key: "7e188fb26e44a0d0fce06a89d7fea484"
server_key_iv: "85ecd6151e1e8ade0b4e431ae771eca9"
server_key_version: "encryptionkey@0"
I20260929 03:00:40.798985 5132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 03:00:40.802506 5147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:40.803928 5132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 03:00:40.804069 5132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "974a0d534419441abc76cb040d722850"
format_stamp: "Formatted at 2026-09-29 03:00:40 on dist-test-slave-q5f0"
server_key: "7e188fb26e44a0d0fce06a89d7fea484"
server_key_iv: "85ecd6151e1e8ade0b4e431ae771eca9"
server_key_version: "encryptionkey@0"
I20260929 03:00:40.804243 5132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:40.819140 5132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:40.823597 5132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:40.823874 5132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:40.841851 5132 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:37293
I20260929 03:00:40.841873 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:37293 every 8 connection(s)
I20260929 03:00:40.843204 5132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:40.847066 5199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:40.849957 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 5132
I20260929 03:00:40.850123 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:40.850553 15433 external_mini_cluster.cc:1428] Setting key 5432a598446e8afad6ca40a3fdd48eae
I20260929 03:00:40.860261 5199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850: Bootstrap starting.
I20260929 03:00:40.864126 5199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:40.865469 5199 log.cc:824] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:40.866051 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:40.852467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57349 (local address 127.15.18.126:37293)
0929 03:00:40.853261 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:40.853277 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:40.853336 (+ 59us) server_negotiation.cc:407] Connection header received
0929 03:00:40.855642 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:40.855674 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:40.856288 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:40.856774 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:40.857319 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:40.859288 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:40.860718 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:40.861728 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:40.861927 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:40.863130 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:40.863141 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:40.863161 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:40.863916 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:40.864017 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:40.864052 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:40.864182 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:40.864408 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 03:00:40.864720 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":352,"threads_started":1}
I20260929 03:00:40.868641 5199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850: No bootstrap required, opened a new log
I20260929 03:00:40.869949 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:40.877084 5199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974a0d534419441abc76cb040d722850" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 37293 } }
I20260929 03:00:40.877604 5199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:40.877720 5199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974a0d534419441abc76cb040d722850, State: Initialized, Role: FOLLOWER
I20260929 03:00:40.878408 5199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974a0d534419441abc76cb040d722850" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 37293 } }
I20260929 03:00:40.878644 5199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:40.878784 5199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:40.878952 5199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:40.880842 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:40.874208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50841 (local address 127.15.18.126:37293)
0929 03:00:40.874435 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:40.874441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:40.874833 (+ 392us) server_negotiation.cc:407] Connection header received
0929 03:00:40.875002 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:40.875006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:40.875071 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:40.875162 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:40.875847 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:40.876587 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:40.878295 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:40.878807 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:40.878931 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:40.880053 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:40.880057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:40.880061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:40.880082 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:40.880141 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:40.880156 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:40.880242 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:40.880405 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 03:00:40.880495 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 03:00:40.881932 5199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974a0d534419441abc76cb040d722850" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 37293 } }
I20260929 03:00:40.882444 5199 leader_election.cc:302] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 974a0d534419441abc76cb040d722850; no voters:
I20260929 03:00:40.882995 5199 leader_election.cc:288] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:40.883109 5209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:40.883739 5209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [term 1 LEADER]: Becoming Leader. State: Replica: 974a0d534419441abc76cb040d722850, State: Running, Role: LEADER
I20260929 03:00:40.884271 5209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974a0d534419441abc76cb040d722850" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 37293 } }
I20260929 03:00:40.885043 5199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:40.886855 5210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "974a0d534419441abc76cb040d722850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974a0d534419441abc76cb040d722850" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 37293 } } }
I20260929 03:00:40.887277 5210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:40.890949 5211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 974a0d534419441abc76cb040d722850. Latest consensus state: current_term: 1 leader_uuid: "974a0d534419441abc76cb040d722850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974a0d534419441abc76cb040d722850" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 37293 } } }
I20260929 03:00:40.891216 5211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:40.892580 5217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:40.897907 5217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:40.907361 5217 catalog_manager.cc:1347] Generated new cluster ID: 79d73f1f340541108b4eebc98a351d65
I20260929 03:00:40.907497 5217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:40.927771 5217 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:40.929342 5217 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:40.934894 5217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 974a0d534419441abc76cb040d722850: Generated new TSK 0
I20260929 03:00:40.935750 5217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:40.943172 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5132
W20260929 03:00:40.944242 5194 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57349
W20260929 03:00:40.944937 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5132: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:40.945065 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 5132
2026-09-29T03:00:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1820 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:41 dist-test-slave-q5f0 krb5kdc[5230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:41 dist-test-slave-q5f0 krb5kdc[5230](info): set up 2 sockets
Sep 29 03:00:41 dist-test-slave-q5f0 krb5kdc[5230](info): commencing operation
krb5kdc: starting...
W20260929 03:00:41.956146 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:42 dist-test-slave-q5f0 krb5kdc[5230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 03:00:42.206585 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43023
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:42.527982 5243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:42.528273 5243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:42.528412 5243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:42.535624 5243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:42.535745 5243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:42.535810 5243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:42.535892 5243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:42.535951 5243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:42.545913 5243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:42.547286 5243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:42.548681 5243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:42.560313 5248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:42.560303 5249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:42.560496 5251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:42.560961 5243 server_base.cc:1034] running on GCE node
I20260929 03:00:42.561726 5243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:42.563127 5243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:42.564356 5243 hybrid_clock.cc:648] HybridClock initialized: now 1790650842564280 us; error 81 us; skew 500 ppm
I20260929 03:00:42.564695 5243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:42 dist-test-slave-q5f0 krb5kdc[5230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650842, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:42.567793 5243 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 03:00:42.569198 5243 webserver.cc:460] Webserver started at http://127.15.18.126:45453/ using document root <none> and password file <none>
I20260929 03:00:42.569856 5243 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:42.569952 5243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:42.570195 5243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:42.573149 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "4515590c3fb84a8d8a96786c454469de"
format_stamp: "Formatted at 2026-09-29 03:00:42 on dist-test-slave-q5f0"
server_key: "572b0c11496340c44b32b8cc3ffeda1d"
server_key_iv: "3a690135c2a8b78d5648da7dcce02471"
server_key_version: "encryptionkey@0"
I20260929 03:00:42.573856 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "4515590c3fb84a8d8a96786c454469de"
format_stamp: "Formatted at 2026-09-29 03:00:42 on dist-test-slave-q5f0"
server_key: "572b0c11496340c44b32b8cc3ffeda1d"
server_key_iv: "3a690135c2a8b78d5648da7dcce02471"
server_key_version: "encryptionkey@0"
I20260929 03:00:42.578836 5243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 03:00:42.582183 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:42.583498 5243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 03:00:42.583694 5243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "4515590c3fb84a8d8a96786c454469de"
format_stamp: "Formatted at 2026-09-29 03:00:42 on dist-test-slave-q5f0"
server_key: "572b0c11496340c44b32b8cc3ffeda1d"
server_key_iv: "3a690135c2a8b78d5648da7dcce02471"
server_key_version: "encryptionkey@0"
I20260929 03:00:42.583863 5243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:42.589190 5243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:42.593034 5243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:42.593266 5243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:42.610281 5243 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:43023
I20260929 03:00:42.610303 5309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:43023 every 8 connection(s)
I20260929 03:00:42.611732 5243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:42.615597 5310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:42.620823 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 5243
I20260929 03:00:42.620977 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:42.621389 15433 external_mini_cluster.cc:1428] Setting key 7d01263b63496aee611892e615d4f037
I20260929 03:00:42.626302 5310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de: Bootstrap starting.
I20260929 03:00:42.628784 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:42.623098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33795 (local address 127.15.18.126:43023)
0929 03:00:42.623912 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:42.623925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:42.623969 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:00:42.625588 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:42.625610 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:42.625964 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:42.626284 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:42.627295 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:42.627302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:42.627314 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:42.627328 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:42.627384 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:42.627403 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:42.627590 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:42.627689 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:00:42.627910 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":380,"threads_started":1}
I20260929 03:00:42.629694 5310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:42.630728 5310 log.cc:824] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:42.632243 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:42.633502 5310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de: No bootstrap required, opened a new log
I20260929 03:00:42.638648 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:42.636305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48185 (local address 127.15.18.126:43023)
0929 03:00:42.636477 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:42.636482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:42.636749 (+ 267us) server_negotiation.cc:407] Connection header received
0929 03:00:42.636887 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:42.636893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:42.636956 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:42.637060 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:42.638024 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:42.638030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:42.638036 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:00:42.638056 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:42.638108 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:00:42.638130 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:42.638248 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:42.638390 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 03:00:42.638440 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 03:00:42.642055 5310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4515590c3fb84a8d8a96786c454469de" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43023 } }
I20260929 03:00:42.642387 5310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:00:42.642465 5310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4515590c3fb84a8d8a96786c454469de, State: Initialized, Role: FOLLOWER
I20260929 03:00:42.643092 5310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4515590c3fb84a8d8a96786c454469de" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43023 } }
I20260929 03:00:42.643359 5310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:00:42.643493 5310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:00:42.643678 5310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:42.646041 5310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4515590c3fb84a8d8a96786c454469de" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43023 } }
I20260929 03:00:42.646615 5310 leader_election.cc:302] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4515590c3fb84a8d8a96786c454469de; no voters:
I20260929 03:00:42.647161 5310 leader_election.cc:288] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:00:42.647285 5320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:42.647943 5320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [term 1 LEADER]: Becoming Leader. State: Replica: 4515590c3fb84a8d8a96786c454469de, State: Running, Role: LEADER
I20260929 03:00:42.648486 5320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4515590c3fb84a8d8a96786c454469de" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43023 } }
I20260929 03:00:42.649207 5310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:42.651104 5322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4515590c3fb84a8d8a96786c454469de. Latest consensus state: current_term: 1 leader_uuid: "4515590c3fb84a8d8a96786c454469de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4515590c3fb84a8d8a96786c454469de" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43023 } } }
I20260929 03:00:42.651472 5322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:42.652158 5321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4515590c3fb84a8d8a96786c454469de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4515590c3fb84a8d8a96786c454469de" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43023 } } }
I20260929 03:00:42.652565 5321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:42.653939 5326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:42.659132 5326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:42.669162 5326 catalog_manager.cc:1347] Generated new cluster ID: 5681ad0693fc4476a8a0f0c827762345
I20260929 03:00:42.669283 5326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:42.687420 5326 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:42.688011 5326 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:42.697180 5326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4515590c3fb84a8d8a96786c454469de: Generated new TSK 0
I20260929 03:00:42.697992 5326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:42.704365 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5243
W20260929 03:00:42.705821 5305 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33795
W20260929 03:00:42.706543 15433 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5243: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 03:00:42.706666 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 5243
2026-09-29T03:00:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1761 ms)
[----------] 18 tests from AuthTokenIssuingTest (33047 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:42 dist-test-slave-q5f0 krb5kdc[5341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:42 dist-test-slave-q5f0 krb5kdc[5341](info): set up 2 sockets
Sep 29 03:00:42 dist-test-slave-q5f0 krb5kdc[5341](info): commencing operation
krb5kdc: starting...
W20260929 03:00:43.724349 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:43 dist-test-slave-q5f0 krb5kdc[5341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 03:00:43.976480 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43725
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43725,127.15.18.125:45769,127.15.18.124:43523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:44.299105 5354 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:00:44.299430 5354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:44.299551 5354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:44.299777 5354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:44.307287 5354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:44.307402 5354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:44.307461 5354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:44.307548 5354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:44.307637 5354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:44.317986 5354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43725,127.15.18.125:45769,127.15.18.124:43523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:44.319334 5354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:44.320634 5354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:44.332690 5359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:44.332650 5360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:44.332711 5362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:44.333321 5354 server_base.cc:1034] running on GCE node
I20260929 03:00:44.334263 5354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:44.336052 5354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:44.337363 5354 hybrid_clock.cc:648] HybridClock initialized: now 1790650844337272 us; error 64 us; skew 500 ppm
I20260929 03:00:44.337736 5354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:44 dist-test-slave-q5f0 krb5kdc[5341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650844, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:44.341423 5354 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 03:00:44.343021 5354 webserver.cc:460] Webserver started at http://127.15.18.126:46075/ using document root <none> and password file <none>
I20260929 03:00:44.343780 5354 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:44.343883 5354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:44.344174 5354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:44.347162 5354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "ff6846909146408391e4dc73de01e9ed"
format_stamp: "Formatted at 2026-09-29 03:00:44 on dist-test-slave-q5f0"
server_key: "4e1c4ac7e8c725f558dfa91fe7715194"
server_key_iv: "5fd53a0b7e9eb323ecfcb0b8291d0252"
server_key_version: "encryptionkey@0"
I20260929 03:00:44.347977 5354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "ff6846909146408391e4dc73de01e9ed"
format_stamp: "Formatted at 2026-09-29 03:00:44 on dist-test-slave-q5f0"
server_key: "4e1c4ac7e8c725f558dfa91fe7715194"
server_key_iv: "5fd53a0b7e9eb323ecfcb0b8291d0252"
server_key_version: "encryptionkey@0"
I20260929 03:00:44.353542 5354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 03:00:44.358381 5369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:44.360247 5354 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 03:00:44.360442 5354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "ff6846909146408391e4dc73de01e9ed"
format_stamp: "Formatted at 2026-09-29 03:00:44 on dist-test-slave-q5f0"
server_key: "4e1c4ac7e8c725f558dfa91fe7715194"
server_key_iv: "5fd53a0b7e9eb323ecfcb0b8291d0252"
server_key_version: "encryptionkey@0"
I20260929 03:00:44.360677 5354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:44.374432 5354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:44.378926 5354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:44.379185 5354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:44.396072 5354 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:43725
I20260929 03:00:44.396080 5420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:43725 every 8 connection(s)
I20260929 03:00:44.397413 5354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:44.400463 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 5354
I20260929 03:00:44.400643 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:44.401254 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:44.401278 15433 external_mini_cluster.cc:1428] Setting key 643660edc2ed0fdf72f58335cd5b7bbe
I20260929 03:00:44.407791 5421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:00:44 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650844, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.18.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.125@KRBTEST.COM" created.
I20260929 03:00:44.429984 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.410939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.15.18.126:43725)
0929 03:00:44.412563 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:44.412571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:44.412618 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:00:44.414132 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:44.414138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:44.414389 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:44.414567 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:44.415904 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.417093 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.418459 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.419021 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.419193 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:44.422522 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:44.422531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:44.422545 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:44.422573 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:44.425638 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:44.426321 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:44.426331 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:44.426334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:44.426481 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:44.426861 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:44.426865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:44.426867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:44.427377 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:44.427594 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:44.428142 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 03:00:44.428441 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1421,"thread_start_us":1214,"threads_started":1}
I20260929 03:00:44.430001 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.410939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43725 (local address 127.0.0.1:35222)
0929 03:00:44.411785 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.411833 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:44.413764 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:44.414606 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:44.414612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:44.415315 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:44.415701 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:44.415726 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.417241 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.417244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:44.418321 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:44.418341 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.419248 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.419255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:44.419484 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:44.420130 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:44.420153 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:44.422296 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:44.425766 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:44.425779 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:44.425793 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:44.426188 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:44.426585 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:44.426590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:44.426594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:44.426748 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:44.427526 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:44.427535 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:44.427760 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 03:00:44.427984 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 03:00:44.428582 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":176,"thread_start_us":340,"threads_started":1}
I20260929 03:00:44.433017 5421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:00:44.434116 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.433723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38858)
0929 03:00:44.433886 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.433998 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 03:00:44.434358 5373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
I20260929 03:00:44.435889 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.435554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38866)
0929 03:00:44.435686 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.435755 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:00:44.436079 5421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:45769: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab.
Entry for principal kudu/127.15.18.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab.
I20260929 03:00:44.454538 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.125:45769
--webserver_interface=127.15.18.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab
--principal=kudu/127.15.18.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43725,127.15.18.125:45769,127.15.18.124:43523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:00:44.485419 5421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } attempt: 1
I20260929 03:00:44.486531 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.486091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38874)
0929 03:00:44.486295 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.486396 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 03:00:44.487553 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.487187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38878)
0929 03:00:44.487322 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.487392 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 03:00:44.487733 5421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:45769: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
I20260929 03:00:44.556005 5421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } attempt: 2
I20260929 03:00:44.557192 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.556697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38888)
0929 03:00:44.556957 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.557050 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 03:00:44.558388 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.558010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38892)
0929 03:00:44.558158 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.558244 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 03:00:44.558601 5421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:45769: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
I20260929 03:00:44.649868 5421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } attempt: 3
I20260929 03:00:44.651161 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.650678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38908)
0929 03:00:44.650923 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.651020 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 03:00:44.652259 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.651886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38920)
0929 03:00:44.652025 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.652101 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:00:44.652468 5421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:45769: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
W20260929 03:00:44.780337 5428 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:00:44.780670 5428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:44.780742 5428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:44.780822 5428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:44.788091 5428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:44.788184 5428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:44.788239 5428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:44.788288 5428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:44.788348 5428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:44.795706 5421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } attempt: 4
I20260929 03:00:44.797003 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.796421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38924)
0929 03:00:44.796760 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.796853 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260929 03:00:44.798069 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.797707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38932)
0929 03:00:44.797852 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.797940 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:00:44.798264 5421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:45769: Network error: Client connection negotiation failed: client connection to 127.15.18.125:45769: connect: Connection refused (error 111)
I20260929 03:00:44.798624 5428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43725,127.15.18.125:45769,127.15.18.124:43523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab
--principal=kudu/127.15.18.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.125:45769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.18.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:44.799916 5428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:44.801149 5428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:44.812785 5434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:44.812865 5433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:44.812772 5436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:44.813726 5428 server_base.cc:1034] running on GCE node
I20260929 03:00:44.814546 5428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:44.816068 5428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:44.817291 5428 hybrid_clock.cc:648] HybridClock initialized: now 1790650844817227 us; error 37 us; skew 500 ppm
I20260929 03:00:44.817606 5428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:44 dist-test-slave-q5f0 krb5kdc[5341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650844, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:44.820806 5428 init.cc:377] Logged in from keytab as kudu/127.15.18.125@KRBTEST.COM (short username kudu)
I20260929 03:00:44.822355 5428 webserver.cc:460] Webserver started at http://127.15.18.125:43153/ using document root <none> and password file <none>
I20260929 03:00:44.823045 5428 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:44.823144 5428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:44.823454 5428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:44.826615 5428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/data/instance:
uuid: "3eee4e60a0c24a6482601825c5c66de5"
format_stamp: "Formatted at 2026-09-29 03:00:44 on dist-test-slave-q5f0"
server_key: "142d5c095ce671423f2979ad6c6ed21f"
server_key_iv: "2b24fe90c5b00217b39ff118150d85e3"
server_key_version: "encryptionkey@0"
I20260929 03:00:44.827416 5428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/wal/instance:
uuid: "3eee4e60a0c24a6482601825c5c66de5"
format_stamp: "Formatted at 2026-09-29 03:00:44 on dist-test-slave-q5f0"
server_key: "142d5c095ce671423f2979ad6c6ed21f"
server_key_iv: "2b24fe90c5b00217b39ff118150d85e3"
server_key_version: "encryptionkey@0"
I20260929 03:00:44.832540 5428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 03:00:44.836022 5443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:44.837443 5428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 03:00:44.837604 5428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/wal
uuid: "3eee4e60a0c24a6482601825c5c66de5"
format_stamp: "Formatted at 2026-09-29 03:00:44 on dist-test-slave-q5f0"
server_key: "142d5c095ce671423f2979ad6c6ed21f"
server_key_iv: "2b24fe90c5b00217b39ff118150d85e3"
server_key_version: "encryptionkey@0"
I20260929 03:00:44.837813 5428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:44.864771 5428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:44.869230 5428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:44.869524 5428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:44.887253 5428 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.125:45769
I20260929 03:00:44.887279 5494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.125:45769 every 8 connection(s)
I20260929 03:00:44.888628 5428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
I20260929 03:00:44.889202 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 5428
I20260929 03:00:44.889353 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-1/wal/instance
I20260929 03:00:44.889935 15433 external_mini_cluster.cc:1428] Setting key 3e07762376cc5b68150353874644f835
I20260929 03:00:44.892489 5495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:44.899005 5495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.18.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.124@KRBTEST.COM" created.
Sep 29 03:00:44 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650844, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 03:00:44.920686 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.901698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.15.18.126:43725)
0929 03:00:44.901915 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:44.901921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:44.903099 (+ 1178us) server_negotiation.cc:407] Connection header received
0929 03:00:44.905042 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:44.905048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:44.905134 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:44.905246 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:44.906723 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.907826 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.909331 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.910075 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.910232 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:44.914839 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:44.914845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:44.914848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:44.914908 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:44.917530 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:44.918347 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:44.918352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:44.918354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:44.918482 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:44.918908 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:44.918912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:44.918914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:44.919197 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:44.919334 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:44.920134 (+ 800us) server_negotiation.cc:299] Negotiation successful
0929 03:00:44.920367 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:00:44.921684 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.902132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43725 (local address 127.0.0.1:35230)
0929 03:00:44.902920 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.902964 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:44.904780 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:44.905454 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:44.905461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:44.905969 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:44.906540 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:44.906559 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.907973 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.907976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:44.909169 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:44.909182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.910295 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.910300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:44.910510 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:44.911139 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:44.911163 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:44.914568 (+ 3405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:44.917702 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:44.917713 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:44.917734 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:44.918220 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:44.918594 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:44.918599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:44.918603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:44.918793 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:44.919335 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:44.919344 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:44.919763 (+ 419us) client_negotiation.cc:769] Sending connection context
0929 03:00:44.920019 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 03:00:44.920580 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":332,"threads_started":1}
I20260929 03:00:44.923578 5495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:00:44 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650844, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
I20260929 03:00:44.939344 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.924343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38940)
0929 03:00:44.924516 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.924530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:44.924619 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:44.925854 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:44.925859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:44.926306 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:44.926483 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:44.926494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.927887 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.927893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:44.929068 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:44.929082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.929885 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.929891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:44.930052 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:44.930643 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:44.930664 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:44.932978 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:44.936878 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:44.936882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:44.936884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:44.937216 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:44.937626 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:44.937630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:44.937634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:44.937870 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:44.938498 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:44.938502 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:44.938675 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 03:00:44.938849 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 03:00:44.939075 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 03:00:44.939666 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.924769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.15.18.125:45769)
0929 03:00:44.925430 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:44.925436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:44.925462 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:00:44.925507 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:44.925513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:44.925687 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:44.925803 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:44.926643 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.927760 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.929195 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:44.929746 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:44.929836 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:44.933205 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:44.933222 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:44.933240 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:44.933278 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:44.936728 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:44.937355 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:44.937363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:44.937365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:44.937487 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:44.938000 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:44.938004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:44.938006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:44.938359 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:44.938533 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:44.939036 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 03:00:44.939290 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":384,"threads_started":1}
I20260929 03:00:44.942216 5495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:00:44.943506 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.943005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60288)
0929 03:00:44.943184 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.943317 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 03:00:44.943861 5446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
I20260929 03:00:44.945353 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.945015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60298)
0929 03:00:44.945163 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.945251 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 03:00:44.945569 5495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:43523: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab.
Entry for principal kudu/127.15.18.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab.
I20260929 03:00:44.946740 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.124:43523
--webserver_interface=127.15.18.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab
--principal=kudu/127.15.18.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43725,127.15.18.125:45769,127.15.18.124:43523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:00:44.994953 5495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } attempt: 1
I20260929 03:00:44.996331 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.995834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60310)
0929 03:00:44.996041 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.996159 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 03:00:44.997426 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:44.997037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60324)
0929 03:00:44.997191 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:44.997308 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 03:00:44.997642 5495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:43523: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
I20260929 03:00:45.065888 5495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } attempt: 2
I20260929 03:00:45.066946 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.066502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60336)
0929 03:00:45.066693 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.066790 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 03:00:45.067920 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.067595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60346)
0929 03:00:45.067728 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.067800 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:00:45.068118 5495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:43523: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
I20260929 03:00:45.097510 5421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } attempt: 5
Sep 29 03:00:45 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650844, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
I20260929 03:00:45.110311 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.098244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38946)
0929 03:00:45.098463 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.098484 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:45.098621 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:45.098994 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:45.098998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:45.099391 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:45.099611 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.099623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.100592 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.100597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.101449 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.101461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.102185 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.102191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.102403 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.102985 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:45.103003 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:45.105023 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:45.108007 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.108013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.108017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.108388 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.108729 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.108734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.108738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.108877 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.109372 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:45.109377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:45.109539 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 03:00:45.109658 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 03:00:45.109966 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 03:00:45.110306 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.098324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38946 (local address 127.15.18.125:45769)
0929 03:00:45.098534 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.098540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.098561 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:00:45.098758 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.098762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.098839 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.098935 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.099769 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.100462 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.101603 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.102043 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.102151 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.105209 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.105214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.105217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.105249 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.107884 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.108498 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.108502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.108504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.108628 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.108975 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.108980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.108983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.109262 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.109402 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.109775 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.109991 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 03:00:45.111706 5421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:00:45.112699 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.112323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60362)
0929 03:00:45.112486 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.112560 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 03:00:45.113557 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.113247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60364)
0929 03:00:45.113384 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.113459 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 03:00:45.113737 5421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:43523: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
I20260929 03:00:45.159332 5495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } attempt: 3
I20260929 03:00:45.160522 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.160047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60378)
0929 03:00:45.160273 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.160365 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 03:00:45.161528 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.161127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60392)
0929 03:00:45.161339 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.161419 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260929 03:00:45.161725 5495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:43523: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
I20260929 03:00:45.164919 5421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } attempt: 1
I20260929 03:00:45.166050 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.165596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60400)
0929 03:00:45.165795 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.165883 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 03:00:45.167160 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.166705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60416)
0929 03:00:45.166938 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.167032 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 03:00:45.167394 5421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:43523: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
I20260929 03:00:45.235623 5421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } attempt: 2
I20260929 03:00:45.236891 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.236341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60422)
0929 03:00:45.236621 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.236713 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 03:00:45.237923 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.237553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60430)
0929 03:00:45.237702 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.237787 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 03:00:45.238139 5421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:43523: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
W20260929 03:00:45.276069 5502 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:00:45.276446 5502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:45.276520 5502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:45.276600 5502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:45.283962 5502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:45.284058 5502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:45.284112 5502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:45.284160 5502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:45.284219 5502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:45.294129 5502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43725,127.15.18.125:45769,127.15.18.124:43523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab
--principal=kudu/127.15.18.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.124:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.18.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:45.295384 5502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:45.296710 5502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:00:45.304921 5495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } attempt: 4
I20260929 03:00:45.306097 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.305551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60444)
0929 03:00:45.305787 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.305908 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 03:00:45.307196 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.306807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60446)
0929 03:00:45.306969 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.307058 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 03:00:45.307540 5495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:43523: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
W20260929 03:00:45.308365 5508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:45.308481 5507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:45.308601 5510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:45.309660 5502 server_base.cc:1034] running on GCE node
I20260929 03:00:45.310377 5502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:45.311779 5502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:45.312995 5502 hybrid_clock.cc:648] HybridClock initialized: now 1790650845312932 us; error 36 us; skew 500 ppm
I20260929 03:00:45.313304 5502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:45 dist-test-slave-q5f0 krb5kdc[5341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650845, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:45.316370 5502 init.cc:377] Logged in from keytab as kudu/127.15.18.124@KRBTEST.COM (short username kudu)
I20260929 03:00:45.317886 5502 webserver.cc:460] Webserver started at http://127.15.18.124:44109/ using document root <none> and password file <none>
I20260929 03:00:45.318593 5502 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:45.318702 5502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:45.319017 5502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:45.322131 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/data/instance:
uuid: "6e5eb3f96f2b43328fb82d67568f75c8"
format_stamp: "Formatted at 2026-09-29 03:00:45 on dist-test-slave-q5f0"
server_key: "e48a5c543516fae5fe57c4f157bce79b"
server_key_iv: "18aef877521eb793b01e723c59e24b6c"
server_key_version: "encryptionkey@0"
I20260929 03:00:45.322925 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/wal/instance:
uuid: "6e5eb3f96f2b43328fb82d67568f75c8"
format_stamp: "Formatted at 2026-09-29 03:00:45 on dist-test-slave-q5f0"
server_key: "e48a5c543516fae5fe57c4f157bce79b"
server_key_iv: "18aef877521eb793b01e723c59e24b6c"
server_key_version: "encryptionkey@0"
I20260929 03:00:45.328701 5502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 03:00:45.332325 5517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:45.333860 5502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:00:45.334025 5502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/wal
uuid: "6e5eb3f96f2b43328fb82d67568f75c8"
format_stamp: "Formatted at 2026-09-29 03:00:45 on dist-test-slave-q5f0"
server_key: "e48a5c543516fae5fe57c4f157bce79b"
server_key_iv: "18aef877521eb793b01e723c59e24b6c"
server_key_version: "encryptionkey@0"
I20260929 03:00:45.334230 5502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:45.344341 5421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } attempt: 3
I20260929 03:00:45.345464 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.344960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60456)
0929 03:00:45.345198 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.345307 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 03:00:45.346462 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.346070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60464)
0929 03:00:45.346272 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.346347 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 03:00:45.346698 5421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:43523: Network error: Client connection negotiation failed: client connection to 127.15.18.124:43523: connect: Connection refused (error 111)
I20260929 03:00:45.349365 5502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:45.353747 5502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:45.353971 5502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:45.371120 5502 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.124:43523
I20260929 03:00:45.371127 5568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.124:43523 every 8 connection(s)
I20260929 03:00:45.372443 5502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
I20260929 03:00:45.376072 5569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:45.381052 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 5502
I20260929 03:00:45.381217 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790650496826590-15433-0/minicluster-data/master-2/wal/instance
I20260929 03:00:45.381652 15433 external_mini_cluster.cc:1428] Setting key cea0767e1f3cd0cfd47deedb7d96cdb1
I20260929 03:00:45.382728 5569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:00:45 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650845, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
Sep 29 03:00:45 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
I20260929 03:00:45.402211 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.385765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.15.18.126:43725)
0929 03:00:45.385970 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.385976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.386671 (+ 695us) server_negotiation.cc:407] Connection header received
0929 03:00:45.387174 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.387179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.387280 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.387392 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.388714 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.389431 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.391112 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.391671 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.391769 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.395689 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.395698 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.395703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.395748 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.399319 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.400111 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.400117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.400124 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.400307 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.400671 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.400676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.400678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.400960 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.401101 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.401615 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.401843 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:00:45.403126 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.385746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43725 (local address 127.0.0.1:35236)
0929 03:00:45.386507 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.386557 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:45.386904 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:45.387471 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:45.387477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:45.388181 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:45.388530 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.388552 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.389596 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.389602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.390968 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.390980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.391924 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.391932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.392306 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.393080 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:45.393103 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:45.395401 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:45.399448 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.399460 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.399474 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.399964 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.400417 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.400422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.400426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.400566 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.401081 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:45.401088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:45.401275 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 03:00:45.401498 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 03:00:45.401967 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":98,"spinlock_wait_cycles":19712,"thread_start_us":358,"threads_started":1}
I20260929 03:00:45.403446 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.383642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.15.18.124:43523)
0929 03:00:45.384433 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.384449 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.384501 (+ 52us) server_negotiation.cc:407] Connection header received
0929 03:00:45.386779 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.386805 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.387097 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.387269 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.388736 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.390065 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.391411 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.392249 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.392386 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.396280 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.396288 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.396301 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.396328 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.398940 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.399697 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.399706 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.399710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.399917 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.400345 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.400350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.400353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.401048 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.401346 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.402747 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.402989 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":550,"thread_start_us":316,"threads_started":1}
I20260929 03:00:45.405452 5569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:00:45 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650845, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
Sep 29 03:00:45 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 03:00:45.418809 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.406104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:45769 (local address 127.0.0.1:38962)
0929 03:00:45.406752 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.406768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:45.406909 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:45.407279 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:45.407284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:45.407737 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:45.407936 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.407949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.408959 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.408964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.409893 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.409904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.410565 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.410571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.410746 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.411273 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:45.411289 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:45.413466 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:45.416498 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.416504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.416509 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.416872 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.417223 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.417229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.417233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.417389 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.417878 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:45.417883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:45.418031 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 03:00:45.418156 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 03:00:45.418445 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551}
I20260929 03:00:45.418825 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.406477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.15.18.125:45769)
0929 03:00:45.406633 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.406638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.406826 (+ 188us) server_negotiation.cc:407] Connection header received
0929 03:00:45.407050 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.407056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.407133 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.407215 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.408116 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.408799 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.410030 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.410436 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.410543 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.413639 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.413644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.413647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.413679 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.416376 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.416982 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.416988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.416992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.417120 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.417488 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.417493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.417496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.417758 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.417907 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.418263 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.418462 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:00:45.419737 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.406391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.15.18.126:43725)
0929 03:00:45.406564 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.406570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.406662 (+ 92us) server_negotiation.cc:407] Connection header received
0929 03:00:45.406834 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.406839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.406922 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.407010 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.408536 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.409277 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.410464 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.410869 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.410973 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.414828 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.414833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.414837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.414867 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.417294 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.417908 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.417914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.417917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.418047 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.418410 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.418414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.418415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.418684 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.418941 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.419206 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.419438 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:00:45.420080 5569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:00:45 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650845, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
Sep 29 03:00:45 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
I20260929 03:00:45.435093 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.421017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.15.18.124:43523)
0929 03:00:45.421160 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.421166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.421195 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:00:45.421242 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.421246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.421315 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.421406 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.422060 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.422989 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.424800 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.425389 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.425509 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.429661 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.429667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.429672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.429736 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.432673 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.433314 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.433319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.433323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.433442 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.433748 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.433753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.433757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.434018 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.434139 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.434492 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.434821 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":44}
I20260929 03:00:45.435070 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.420846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60482)
0929 03:00:45.421019 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.421038 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:45.421132 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:45.421440 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:45.421446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:45.421791 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:45.421940 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.421948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.423196 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.423201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.424644 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.424660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.425560 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.425566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.425906 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.426767 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:45.426789 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:45.429345 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:45.432821 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.432825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.432829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.433204 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.433535 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.433539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.433543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.433659 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.434120 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:45.434125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:45.434269 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 03:00:45.434396 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 03:00:45.434667 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 03:00:45.437559 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.421406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.15.18.125:45769)
0929 03:00:45.422124 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.422128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.422140 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:00:45.422176 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.422179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.422228 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.422294 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.423905 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.424963 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.426334 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.426884 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.427000 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.432268 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.432273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.432276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.432309 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.434708 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.435402 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.435408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.435412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.435589 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.435992 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.435997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.436000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.436339 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.436567 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.436972 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.437190 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627}
I20260929 03:00:45.441151 5569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8: Bootstrap starting.
I20260929 03:00:45.442922 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:45.444713 5569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:45.446012 5569 log.cc:824] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:45.445511 5396 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ff6846909146408391e4dc73de01e9ed"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35240
I20260929 03:00:45.449317 5569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8: No bootstrap required, opened a new log
I20260929 03:00:45.460099 5569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } }
I20260929 03:00:45.460556 5569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5eb3f96f2b43328fb82d67568f75c8, State: Initialized, Role: FOLLOWER
I20260929 03:00:45.461069 5569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } }
I20260929 03:00:45.462041 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } } }
I20260929 03:00:45.462375 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:45.463160 5569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:00:45.472015 5586 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:00:45.472144 5586 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:00:45.469982 5502 master_runner.cc:186] Error getting master registration for 127.15.18.126:43725: OK, instance_id { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" instance_seqno: 1790650844391030 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:00:45.480204 5502 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:00:45.523957 5421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } attempt: 4
Sep 29 03:00:45 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650844, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
I20260929 03:00:45.541014 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.524969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.15.18.124:43523)
0929 03:00:45.525185 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.525193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.525354 (+ 161us) server_negotiation.cc:407] Connection header received
0929 03:00:45.525495 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.525502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.525600 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.525778 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.526780 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.528025 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.529906 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.530409 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.530525 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.534546 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.534554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.534558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.534594 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.538084 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.538754 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.538760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.538765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.538906 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.539375 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.539380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.539384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.539870 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.540016 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.540418 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.540645 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 03:00:45.541033 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.524889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60486)
0929 03:00:45.525156 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.525180 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:45.525354 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:45.525810 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:45.525815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:45.526374 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:45.526600 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.526615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.528216 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.528222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.529760 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.529778 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.530570 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.530576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.530786 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.531548 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:45.531575 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:45.534340 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:45.538232 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.538239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.538243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.538632 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.539018 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.539023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.539027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.539261 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.540002 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:45.540006 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:45.540167 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 03:00:45.540292 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 03:00:45.540585 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 03:00:45.546764 5421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed: Bootstrap starting.
I20260929 03:00:45.550909 5421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:45.552284 5421 log.cc:824] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:45.555114 5421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed: No bootstrap required, opened a new log
I20260929 03:00:45.560274 5421 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } }
I20260929 03:00:45.560585 5421 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6846909146408391e4dc73de01e9ed, State: Initialized, Role: FOLLOWER
I20260929 03:00:45.561066 5421 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } }
I20260929 03:00:45.561969 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } } }
I20260929 03:00:45.562318 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:45.563164 5421 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:45.563313 5396 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:00:45.563534 5396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:45.566968 5396 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } }
I20260929 03:00:45.569522 5544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff6846909146408391e4dc73de01e9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6e5eb3f96f2b43328fb82d67568f75c8"
I20260929 03:00:45.570007 5544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:45.573578 5544 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6846909146408391e4dc73de01e9ed in term 1.
I20260929 03:00:45.574389 5372 leader_election.cc:302] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5eb3f96f2b43328fb82d67568f75c8, ff6846909146408391e4dc73de01e9ed; no voters:
I20260929 03:00:45.574944 5589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:45.575644 5589 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [term 1 LEADER]: Becoming Leader. State: Replica: ff6846909146408391e4dc73de01e9ed, State: Running, Role: LEADER
I20260929 03:00:45.576267 5589 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } }
I20260929 03:00:45.580355 5396 leader_election.cc:288] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [CANDIDATE]: Term 1 election: Requested vote from peers 3eee4e60a0c24a6482601825c5c66de5 (127.15.18.125:45769), 6e5eb3f96f2b43328fb82d67568f75c8 (127.15.18.124:43523)
I20260929 03:00:45.580053 5595 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff6846909146408391e4dc73de01e9ed. Latest consensus state: current_term: 1 leader_uuid: "ff6846909146408391e4dc73de01e9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } } }
I20260929 03:00:45.581054 5595 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:45.597954 5597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 03:00:45.606065 5523 tablet.cc:2369] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:00:45.606745 5495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } attempt: 5
I20260929 03:00:45.625123 5597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:00:45 dist-test-slave-q5f0 krb5kdc[5341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650844, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
I20260929 03:00:45.636837 5544 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [term 1 FOLLOWER]: Refusing update from remote peer ff6846909146408391e4dc73de01e9ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 03:00:45.643733 5375 tablet.cc:2369] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:00:45.644726 5595 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260929 03:00:45.651539 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.599618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.15.18.125:45769)
0929 03:00:45.599810 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.599816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.616140 (+ 16324us) server_negotiation.cc:407] Connection header received
0929 03:00:45.616411 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.616418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.616505 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.616580 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.620068 (+ 3488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.621171 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.634947 (+ 13776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.635598 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.635710 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.639369 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.639377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.639381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.639423 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.643827 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.645577 (+ 1750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.645583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.645587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.645869 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.646732 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.646766 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.646770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.647385 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.647910 (+ 525us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.649282 (+ 1372us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.649913 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:00:45.654955 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.607432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:43523 (local address 127.0.0.1:60510)
0929 03:00:45.607800 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:45.607818 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:45.607945 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:45.619815 (+ 11870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:45.619821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:45.620393 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:45.620599 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.620612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.622050 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.622056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.623436 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:45.623452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.625910 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.625916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:45.626130 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.626778 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:45.626800 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:45.632051 (+ 5251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:45.643406 (+ 11355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.643412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.643417 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.652274 (+ 8857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.652976 (+ 702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:45.652982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:45.652985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:45.653156 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:45.653970 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:45.653975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:45.654138 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 03:00:45.654291 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 03:00:45.654590 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260929 03:00:45.656728 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.608148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.15.18.124:43523)
0929 03:00:45.619478 (+ 11330us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.619483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.619523 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:00:45.619580 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.619585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.619659 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.619769 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.620761 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.621914 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.623647 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.625668 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.626091 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.632290 (+ 6199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.632297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.632302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.632342 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.643176 (+ 10834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.652453 (+ 9277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.652462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.652466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.652681 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.653339 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.653344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.653348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.653710 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.653871 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.655411 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.655864 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11194,"thread_start_us":10995,"threads_started":1}
I20260929 03:00:45.657137 5612 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:00:45.659094 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.599622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.15.18.124:43523)
0929 03:00:45.599808 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.599814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.603743 (+ 3929us) server_negotiation.cc:407] Connection header received
0929 03:00:45.604268 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.604276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.604406 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.604488 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.611557 (+ 7069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.612910 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.631547 (+ 18637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.632251 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.632339 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.649067 (+ 16728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.649074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.649080 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.649119 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.652729 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.655671 (+ 2942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.655676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.655680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.655822 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.657312 (+ 1490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.657317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.657320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.657698 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.657957 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.658520 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.658757 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 03:00:45.633170 5354 master_runner.cc:186] Error getting master registration for 127.15.18.125:45769: OK, instance_id { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" instance_seqno: 1790650844882288 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:00:45.661937 5354 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:00:45.662758 5495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5: Bootstrap starting.
I20260929 03:00:45.670152 5495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:45.672811 5495 log.cc:824] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:45.674113 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.600177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.15.18.126:43725)
0929 03:00:45.600341 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.600346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.629544 (+ 29198us) server_negotiation.cc:407] Connection header received
0929 03:00:45.631006 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.631011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.631091 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.631154 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.643635 (+ 12481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.644711 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.647238 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.647890 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.647991 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.662268 (+ 14277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:45.662275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:45.662280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:45.662328 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:45.666337 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.667293 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.667299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.667302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.667484 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:45.671704 (+ 4220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:45.671713 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:45.671718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:45.672228 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:45.672521 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.673428 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.673711 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:00:45.674335 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff6846909146408391e4dc73de01e9ed. Latest consensus state: current_term: 1 leader_uuid: "ff6846909146408391e4dc73de01e9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } } }
I20260929 03:00:45.674688 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:45.687973 5495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5: No bootstrap required, opened a new log
I20260929 03:00:45.691901 5614 mvcc.cc:205] Tried to move back new op lower bound from 7334505863712079872 to 7334505863485841408. Current Snapshot: MvccSnapshot[applied={T|T < 7334505863712079872}]
I20260929 03:00:45.707267 5495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } }
I20260929 03:00:45.707729 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff6846909146408391e4dc73de01e9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } } }
I20260929 03:00:45.707969 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:45.708025 5495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eee4e60a0c24a6482601825c5c66de5, State: Initialized, Role: FOLLOWER
I20260929 03:00:45.708949 5495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } }
I20260929 03:00:45.714037 5597 catalog_manager.cc:1347] Generated new cluster ID: f4787162757a45d4a7c19629d1f5fb1e
I20260929 03:00:45.714150 5597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:45.715380 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff6846909146408391e4dc73de01e9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } } }
I20260929 03:00:45.716210 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:45.720131 5619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } } }
I20260929 03:00:45.720542 5619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:45.722325 5470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff6846909146408391e4dc73de01e9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3eee4e60a0c24a6482601825c5c66de5"
I20260929 03:00:45.722661 5469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:45.722971 5495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:45.728271 5469 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [term 1 FOLLOWER]: Refusing update from remote peer ff6846909146408391e4dc73de01e9ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:00:45.731887 5470 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6846909146408391e4dc73de01e9ed in term 1.
I20260929 03:00:45.736083 5589 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:00:45.749426 5597 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:45.751431 5597 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:45.785526 5619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff6846909146408391e4dc73de01e9ed. Latest consensus state: current_term: 1 leader_uuid: "ff6846909146408391e4dc73de01e9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } } }
I20260929 03:00:45.785869 5619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:45.786141 5597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed: Generated new TSK 0
I20260929 03:00:45.787181 5597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:45.788645 5629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff6846909146408391e4dc73de01e9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } } }
I20260929 03:00:45.789206 5629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:45.790385 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff6846909146408391e4dc73de01e9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6846909146408391e4dc73de01e9ed" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43725 } } peers { permanent_uuid: "3eee4e60a0c24a6482601825c5c66de5" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 45769 } } peers { permanent_uuid: "6e5eb3f96f2b43328fb82d67568f75c8" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 43523 } } }
I20260929 03:00:45.790650 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6846909146408391e4dc73de01e9ed [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:45.817353 5634 catalog_manager.cc:1259] Loaded cluster ID: f4787162757a45d4a7c19629d1f5fb1e
I20260929 03:00:45.817544 5634 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5: loading cluster ID for follower catalog manager: success
I20260929 03:00:45.821427 5634 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5: acquiring CA information for follower catalog manager: success
I20260929 03:00:45.824136 5634 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3eee4e60a0c24a6482601825c5c66de5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:00:45.933874 5425 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:45.908914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.15.18.126:43725)
0929 03:00:45.909116 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.909122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.930246 (+ 21124us) server_negotiation.cc:407] Connection header received
0929 03:00:45.932186 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.932193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.932282 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.932360 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.933595 (+ 1235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35268: BlockingRecv error: recv got EOF from 127.0.0.1:35268 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 03:00:45.939826 5500 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:45.926621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.15.18.125:45769)
0929 03:00:45.926840 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.926847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.937121 (+ 10274us) server_negotiation.cc:407] Connection header received
0929 03:00:45.937322 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.937329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.937417 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.937533 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.939568 (+ 2035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38990: BlockingRecv error: recv got EOF from 127.0.0.1:38990 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 03:00:45.941092 5572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:45.915246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.15.18.124:43523)
0929 03:00:45.915436 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.915442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.936532 (+ 21090us) server_negotiation.cc:407] Connection header received
0929 03:00:45.938615 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.938621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.938734 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.938804 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:45.940808 (+ 2004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60526: BlockingRecv error: recv got EOF from 127.0.0.1:60526 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:00:45.984530 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:45.969595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39004 (local address 127.15.18.125:45769)
0929 03:00:45.969798 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:45.969805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:45.976679 (+ 6874us) server_negotiation.cc:407] Connection header received
0929 03:00:45.977246 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:45.977252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:45.977471 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:45.977618 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:45.978346 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.979548 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.981498 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:45.982196 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:45.982291 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:45.983007 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:45.983740 (+ 733us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:45.983858 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:45.984090 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 03:00:45.984192 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 03:00:46.027763 5572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:46.017825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.15.18.124:43523)
0929 03:00:46.018065 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.018072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.025511 (+ 7439us) server_negotiation.cc:407] Connection header received
0929 03:00:46.026134 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.026140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.026278 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.026356 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:46.027485 (+ 1129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60528: BlockingRecv error: recv got EOF from 127.0.0.1:60528 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 03:00:46.063061 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.056319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.15.18.125:45769)
0929 03:00:46.056509 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.056515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.057046 (+ 531us) server_negotiation.cc:407] Connection header received
0929 03:00:46.057220 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.057226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.057342 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.057442 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.057914 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.059081 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.060985 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.061681 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.061821 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.062277 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.062420 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.062507 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.062660 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.062754 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 03:00:46.075974 5572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:46.073395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.15.18.124:43523)
0929 03:00:46.073586 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.073592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.074168 (+ 576us) server_negotiation.cc:407] Connection header received
0929 03:00:46.074370 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.074376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.074475 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.074579 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:46.075714 (+ 1135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60534: BlockingRecv error: recv got EOF from 127.0.0.1:60534 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:00:46.088485 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.081777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.15.18.125:45769)
0929 03:00:46.081964 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.081970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.082517 (+ 547us) server_negotiation.cc:407] Connection header received
0929 03:00:46.082743 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.082750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.082836 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.082926 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.083353 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.084442 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.086390 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.087013 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.087133 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.087666 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.087824 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.087915 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.088071 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.088161 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 03:00:46.097225 5572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:46.095260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.15.18.124:43523)
0929 03:00:46.095452 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.095458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.095616 (+ 158us) server_negotiation.cc:407] Connection header received
0929 03:00:46.095939 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.095945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.096044 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.096151 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:46.096996 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60542: BlockingRecv error: recv got EOF from 127.0.0.1:60542 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 03:00:46.120406 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.113265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.15.18.125:45769)
0929 03:00:46.113512 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.113518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.114099 (+ 581us) server_negotiation.cc:407] Connection header received
0929 03:00:46.114306 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.114312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.114409 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.114523 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.115038 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.116243 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.118182 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.118908 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.119051 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.119588 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.119738 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.119844 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.120010 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.120103 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260929 03:00:46.135419 5572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:46.129590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.15.18.124:43523)
0929 03:00:46.129880 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.129886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.133837 (+ 3951us) server_negotiation.cc:407] Connection header received
0929 03:00:46.134033 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.134039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.134141 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.134262 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:46.135152 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60558: BlockingRecv error: recv got EOF from 127.0.0.1:60558 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
I20260929 03:00:46.157090 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.149782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.15.18.125:45769)
0929 03:00:46.149989 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.149996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.150573 (+ 577us) server_negotiation.cc:407] Connection header received
0929 03:00:46.150822 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.150828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.150924 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.151031 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.151604 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.152817 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.154861 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.155596 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.155726 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.156242 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.156402 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.156491 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.156665 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.156760 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 03:00:46.167145 5572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:46.165113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.15.18.124:43523)
0929 03:00:46.165305 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.165311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.165337 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:00:46.165676 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.165682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.165790 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.165895 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:46.166931 (+ 1036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60562: BlockingRecv error: recv got EOF from 127.0.0.1:60562 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 03:00:46.195123 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.188117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.15.18.125:45769)
0929 03:00:46.188354 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.188361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.188889 (+ 528us) server_negotiation.cc:407] Connection header received
0929 03:00:46.189125 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.189131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.189232 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.189355 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.189820 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.191017 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.193004 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.193662 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.193828 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.194298 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.194450 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.194553 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.194726 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.194829 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 03:00:46.210654 5572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:46.208250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.15.18.124:43523)
0929 03:00:46.208465 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.208471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.209030 (+ 559us) server_negotiation.cc:407] Connection header received
0929 03:00:46.209276 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.209282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.209391 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.209519 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:46.210400 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60572: BlockingRecv error: recv got EOF from 127.0.0.1:60572 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 03:00:46.253635 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.246589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.15.18.125:45769)
0929 03:00:46.246795 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.246801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.247395 (+ 594us) server_negotiation.cc:407] Connection header received
0929 03:00:46.247669 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.247675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.247806 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.247919 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.248523 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.249652 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.251662 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.252169 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.252276 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.252833 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.252955 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.253048 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.253212 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.253295 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 03:00:46.261672 5572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:46.259729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.15.18.124:43523)
0929 03:00:46.259910 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.259916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.260293 (+ 377us) server_negotiation.cc:407] Connection header received
0929 03:00:46.260491 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.260495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.260571 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.260655 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:46.261434 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60574: BlockingRecv error: recv got EOF from 127.0.0.1:60574 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:00:46.334426 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.329307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.15.18.125:45769)
0929 03:00:46.329565 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.329572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.329782 (+ 210us) server_negotiation.cc:407] Connection header received
0929 03:00:46.330059 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.330063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.330140 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.330228 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.330732 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.331545 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.332817 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.333242 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.333345 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.333863 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.333979 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.334046 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.334153 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.334219 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260929 03:00:46.342000 5572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:46.340180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.15.18.124:43523)
0929 03:00:46.340348 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.340353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.340718 (+ 365us) server_negotiation.cc:407] Connection header received
0929 03:00:46.340919 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.340923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.340997 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.341086 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:46.341788 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60578: BlockingRecv error: recv got EOF from 127.0.0.1:60578 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:00:46.474395 5586 catalog_manager.cc:1259] Loaded cluster ID: f4787162757a45d4a7c19629d1f5fb1e
I20260929 03:00:46.474499 5586 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8: loading cluster ID for follower catalog manager: success
I20260929 03:00:46.477933 5586 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8: acquiring CA information for follower catalog manager: success
I20260929 03:00:46.479589 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.473642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.15.18.125:45769)
0929 03:00:46.473856 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.473862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.474184 (+ 322us) server_negotiation.cc:407] Connection header received
0929 03:00:46.474418 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.474423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.474496 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.474597 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.475093 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.475861 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.477793 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.478239 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.478355 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.478891 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.478999 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.479066 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.479230 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.479317 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 03:00:46.480278 5586 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e5eb3f96f2b43328fb82d67568f75c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:00:46.490501 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.485051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.15.18.124:43523)
0929 03:00:46.485288 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.485295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.485480 (+ 185us) server_negotiation.cc:407] Connection header received
0929 03:00:46.485610 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.485614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.485763 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.485858 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.486255 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.486988 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.488360 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.488778 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.488872 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.489358 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.489914 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.490031 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.490160 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.490232 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 03:00:46.514882 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.497652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.15.18.124:43523)
0929 03:00:46.498243 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.498249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.508795 (+ 10546us) server_negotiation.cc:407] Connection header received
0929 03:00:46.509119 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.509123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.509197 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.509281 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.510165 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.511080 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.512928 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.513374 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.513503 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.513999 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.514147 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.514245 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.514391 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.514487 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454}
I20260929 03:00:46.516058 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.497859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.15.18.125:45769)
0929 03:00:46.498073 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.498080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.510114 (+ 12034us) server_negotiation.cc:407] Connection header received
0929 03:00:46.510403 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.510410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.510494 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.510580 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.511175 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.512019 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.514099 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.514671 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.514783 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.515253 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.515395 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.515463 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.515710 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.515793 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 03:00:46.517099 5743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:46.497300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.15.18.126:43725)
0929 03:00:46.507919 (+ 10619us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:46.507927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:46.509259 (+ 1332us) server_negotiation.cc:407] Connection header received
0929 03:00:46.509534 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:46.509539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:46.509763 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:46.509950 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:46.510620 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.511740 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.514055 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:46.514716 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:46.514852 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:46.515614 (+ 762us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:46.516080 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:46.516454 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:46.516702 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 03:00:46.516805 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10437,"thread_start_us":10622,"threads_started":1}
I20260929 03:00:46.522928 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5354
I20260929 03:00:46.524031 5416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:46.633569 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 5354
I20260929 03:00:46.650303 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5428
I20260929 03:00:46.651430 5490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:46.758957 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 5428
I20260929 03:00:46.775853 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5502
I20260929 03:00:46.776939 5564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:46.884979 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 5502
2026-09-29T03:00:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4182 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:00:46 dist-test-slave-q5f0 krb5kdc[5749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:00:46 dist-test-slave-q5f0 krb5kdc[5749](info): set up 2 sockets
Sep 29 03:00:46 dist-test-slave-q5f0 krb5kdc[5749](info): commencing operation
krb5kdc: starting...
W20260929 03:00:47.912338 15433 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/test-user.keytab.
Sep 29 03:00:48 dist-test-slave-q5f0 krb5kdc[5749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.18.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
Entry for principal kudu/127.15.18.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab.
I20260929 03:00:48.166870 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43015
--webserver_interface=127.15.18.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43015,127.15.18.125:36687,127.15.18.124:38861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:00:48.490577 5762 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:00:48.490947 5762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:48.491042 5762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:48.491158 5762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:48.498523 5762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:48.498636 5762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:48.498719 5762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:48.498783 5762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:48.498869 5762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:48.509040 5762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43015,127.15.18.125:36687,127.15.18.124:38861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.126.keytab
--principal=kudu/127.15.18.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.126:43015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.18.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:48.510419 5762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:48.511929 5762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:48.524324 5767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:48.524258 5770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:48.524361 5768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:48.525373 5762 server_base.cc:1034] running on GCE node
I20260929 03:00:48.526172 5762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:48.527652 5762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:48.528898 5762 hybrid_clock.cc:648] HybridClock initialized: now 1790650848528831 us; error 29 us; skew 500 ppm
I20260929 03:00:48.529237 5762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:48 dist-test-slave-q5f0 krb5kdc[5749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650848, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:48.532457 5762 init.cc:377] Logged in from keytab as kudu/127.15.18.126@KRBTEST.COM (short username kudu)
I20260929 03:00:48.534067 5762 webserver.cc:460] Webserver started at http://127.15.18.126:46111/ using document root <none> and password file <none>
I20260929 03:00:48.534742 5762 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:48.534835 5762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:48.535137 5762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:48.538125 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/data/instance:
uuid: "1240553dc9d44fbcb26de6ee8e383f58"
format_stamp: "Formatted at 2026-09-29 03:00:48 on dist-test-slave-q5f0"
server_key: "fcbe085af582a2d0a2b75d818dc960a7"
server_key_iv: "347dc5331cd0201dd521cdb0f0e0d08d"
server_key_version: "encryptionkey@0"
I20260929 03:00:48.538930 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/wal/instance:
uuid: "1240553dc9d44fbcb26de6ee8e383f58"
format_stamp: "Formatted at 2026-09-29 03:00:48 on dist-test-slave-q5f0"
server_key: "fcbe085af582a2d0a2b75d818dc960a7"
server_key_iv: "347dc5331cd0201dd521cdb0f0e0d08d"
server_key_version: "encryptionkey@0"
I20260929 03:00:48.544180 5762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260929 03:00:48.547642 5777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:48.549023 5762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 03:00:48.549199 5762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/wal
uuid: "1240553dc9d44fbcb26de6ee8e383f58"
format_stamp: "Formatted at 2026-09-29 03:00:48 on dist-test-slave-q5f0"
server_key: "fcbe085af582a2d0a2b75d818dc960a7"
server_key_iv: "347dc5331cd0201dd521cdb0f0e0d08d"
server_key_version: "encryptionkey@0"
I20260929 03:00:48.549420 5762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:48.561107 5762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:48.565049 5762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:48.565300 5762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:48.582659 5762 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.126:43015
I20260929 03:00:48.582679 5828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.126:43015 every 8 connection(s)
I20260929 03:00:48.584057 5762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/data/info.pb
I20260929 03:00:48.587759 5829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:48.591054 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 5762
I20260929 03:00:48.591251 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-0/wal/instance
I20260929 03:00:48.591797 15433 external_mini_cluster.cc:1428] Setting key d6942270dfa888fa889d77aba7e34a8d
I20260929 03:00:48.596101 5829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:00:48 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650848, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.18.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.125@KRBTEST.COM" created.
I20260929 03:00:48.619122 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.599923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43015 (local address 127.0.0.1:57154)
0929 03:00:48.600814 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.600863 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:48.602900 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:48.603795 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:48.603801 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:48.604503 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:48.604843 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:48.604862 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:48.606352 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:48.606355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:48.607418 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:48.607429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:48.608340 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:48.608347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:48.608506 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:48.609208 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:48.609239 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:48.611668 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:48.614926 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:48.614938 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:48.614951 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:48.615334 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:48.615746 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:48.615751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:48.615755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:48.615925 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:48.616680 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:48.616687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:48.616896 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 03:00:48.617122 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 03:00:48.617775 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":183,"thread_start_us":365,"threads_started":1}
I20260929 03:00:48.619139 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.599923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57154 (local address 127.15.18.126:43015)
0929 03:00:48.601631 (+ 1708us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:48.601640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:48.601695 (+ 55us) server_negotiation.cc:407] Connection header received
0929 03:00:48.603284 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:48.603290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:48.603570 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:48.603767 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:48.605033 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:48.606215 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:48.607572 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:48.608108 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:48.608302 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:48.612036 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:48.612045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:48.612059 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:48.612094 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:48.614793 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:48.615462 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:48.615469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:48.615472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:48.615641 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:48.616022 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:48.616027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:48.616031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:48.616534 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:48.616721 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:48.617256 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 03:00:48.617646 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":1491,"thread_start_us":1275,"threads_started":1}
I20260929 03:00:48.622617 5829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:00:48.623735 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.623271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:50984)
0929 03:00:48.623452 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.623591 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 03:00:48.623962 5781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
I20260929 03:00:48.625423 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.625088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:50988)
0929 03:00:48.625212 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.625287 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 03:00:48.625698 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:36687: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab.
Entry for principal kudu/127.15.18.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab.
I20260929 03:00:48.646826 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.125:36687
--webserver_interface=127.15.18.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab
--principal=kudu/127.15.18.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43015,127.15.18.125:36687,127.15.18.124:38861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:00:48.675041 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } attempt: 1
I20260929 03:00:48.676343 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.675799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:50998)
0929 03:00:48.676021 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.676176 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 03:00:48.677515 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.677067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:51004)
0929 03:00:48.677243 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.677340 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 03:00:48.677758 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:36687: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
I20260929 03:00:48.746034 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } attempt: 2
I20260929 03:00:48.747334 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.746738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:51008)
0929 03:00:48.747046 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.747165 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 03:00:48.748474 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.748081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:51014)
0929 03:00:48.748218 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.748317 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 03:00:48.748659 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:36687: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
I20260929 03:00:48.840003 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } attempt: 3
I20260929 03:00:48.841331 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.840797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:51024)
0929 03:00:48.841043 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.841157 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 03:00:48.842535 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.842113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:51034)
0929 03:00:48.842272 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.842366 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 03:00:48.842774 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:36687: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
W20260929 03:00:48.971719 5836 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:00:48.972075 5836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:48.972179 5836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:48.972297 5836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:48.979620 5836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:48.979717 5836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:48.979776 5836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:48.979842 5836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:48.979933 5836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:48.986027 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } attempt: 4
I20260929 03:00:48.987282 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.986691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:51046)
0929 03:00:48.987010 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.987119 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 03:00:48.988363 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:48.987981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:51060)
0929 03:00:48.988121 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:48.988215 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 03:00:48.988583 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.125:36687: Network error: Client connection negotiation failed: client connection to 127.15.18.125:36687: connect: Connection refused (error 111)
I20260929 03:00:48.990563 5836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43015,127.15.18.125:36687,127.15.18.124:38861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.125.keytab
--principal=kudu/127.15.18.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.125:36687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.18.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:48.991899 5836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:48.993203 5836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:49.004858 5841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:49.004930 5844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:49.004880 5842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:49.005461 5836 server_base.cc:1034] running on GCE node
I20260929 03:00:49.006256 5836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:49.007710 5836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:49.008962 5836 hybrid_clock.cc:648] HybridClock initialized: now 1790650849008867 us; error 38 us; skew 500 ppm
I20260929 03:00:49.009297 5836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650849, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:49.016788 5836 init.cc:377] Logged in from keytab as kudu/127.15.18.125@KRBTEST.COM (short username kudu)
I20260929 03:00:49.018375 5836 webserver.cc:460] Webserver started at http://127.15.18.125:41335/ using document root <none> and password file <none>
I20260929 03:00:49.019098 5836 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:49.019191 5836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:49.019493 5836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:49.022608 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/data/instance:
uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62"
format_stamp: "Formatted at 2026-09-29 03:00:49 on dist-test-slave-q5f0"
server_key: "a3b1f2de65fa08ba65b9948d404fb63b"
server_key_iv: "4968cbf15962f988b6134bea321a8fbd"
server_key_version: "encryptionkey@0"
I20260929 03:00:49.023419 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/wal/instance:
uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62"
format_stamp: "Formatted at 2026-09-29 03:00:49 on dist-test-slave-q5f0"
server_key: "a3b1f2de65fa08ba65b9948d404fb63b"
server_key_iv: "4968cbf15962f988b6134bea321a8fbd"
server_key_version: "encryptionkey@0"
I20260929 03:00:49.028959 5836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 03:00:49.032701 5851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:49.034123 5836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260929 03:00:49.034296 5836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/wal
uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62"
format_stamp: "Formatted at 2026-09-29 03:00:49 on dist-test-slave-q5f0"
server_key: "a3b1f2de65fa08ba65b9948d404fb63b"
server_key_iv: "4968cbf15962f988b6134bea321a8fbd"
server_key_version: "encryptionkey@0"
I20260929 03:00:49.034510 5836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:49.054514 5836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:49.059240 5836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:49.059558 5836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:49.080674 5836 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.125:36687
I20260929 03:00:49.080647 5902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.125:36687 every 8 connection(s)
I20260929 03:00:49.082512 5836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/data/info.pb
I20260929 03:00:49.086804 5903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:49.091246 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 5836
I20260929 03:00:49.091442 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-1/wal/instance
I20260929 03:00:49.092052 15433 external_mini_cluster.cc:1428] Setting key 899bd8f44fd022904f93bea76a659c11
I20260929 03:00:49.093864 5903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650849, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 03:00:49.117654 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.096694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57156 (local address 127.15.18.126:43015)
0929 03:00:49.096916 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.096922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.098160 (+ 1238us) server_negotiation.cc:407] Connection header received
0929 03:00:49.100679 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.100686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.100788 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.100908 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.103367 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.104529 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.106445 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.107041 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.107159 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.110759 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.110765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.110769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.110823 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.114226 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.115052 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.115057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.115061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.115214 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.115721 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.115726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.115729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.116124 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.116290 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.117070 (+ 780us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.117315 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:00:49.118505 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.097155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43015 (local address 127.0.0.1:57156)
0929 03:00:49.097990 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.098041 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:49.100351 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:49.101394 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:49.101409 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:49.102238 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:49.103124 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.103151 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.104751 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.104755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.106286 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.106299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.107238 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.107242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.107473 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.108150 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:49.108172 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:49.110502 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:49.114405 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.114414 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.114429 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.114920 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.115368 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.115374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.115376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.115613 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.116294 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:49.116301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:49.116686 (+ 385us) client_negotiation.cc:769] Sending connection context
0929 03:00:49.116920 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 03:00:49.117325 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":300,"threads_started":1}
WARNING: no policy specified for kudu/127.15.18.124@KRBTEST.COM; defaulting to no policy
I20260929 03:00:49.120254 5903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.18.124@KRBTEST.COM" created.
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650849, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
I20260929 03:00:49.135597 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.121035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:51066)
0929 03:00:49.121234 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.121254 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:49.121339 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:49.122337 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:49.122342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:49.122800 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:49.122972 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.122982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.124398 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.124403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.125475 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.125488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.126271 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.126280 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.126454 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.127016 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:49.127037 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:49.129203 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:49.132970 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.132975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.132979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.133463 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.133817 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.133822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.133826 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.133983 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.134561 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:49.134565 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:49.134722 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 03:00:49.134870 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 03:00:49.135153 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 03:00:49.135670 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.121187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.15.18.125:36687)
0929 03:00:49.121846 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.121853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.121881 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:00:49.121935 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.121942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.122176 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.122325 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.123133 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.124253 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.125602 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.126134 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.126224 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.129425 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.129450 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.129468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.129503 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.132832 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.133595 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.133602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.133604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.133721 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.134078 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.134081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.134083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.134440 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.134588 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.134989 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.135267 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":517,"thread_start_us":376,"threads_started":1}
I20260929 03:00:49.138231 5903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:00:49.139420 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.138864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45218)
0929 03:00:49.139050 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.139207 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 03:00:49.139730 5854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
I20260929 03:00:49.141220 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.140862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45232)
0929 03:00:49.141001 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.141094 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 03:00:49.141498 5903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:38861: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.18.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab.
Entry for principal kudu/127.15.18.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab.
I20260929 03:00:49.155123 15433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
/tmp/dist-test-taskxb3wIY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.124:38861
--webserver_interface=127.15.18.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab
--principal=kudu/127.15.18.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.18.84:46563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.18.126:43015,127.15.18.125:36687,127.15.18.124:38861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:00:49.190846 5903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } attempt: 1
I20260929 03:00:49.192183 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.191627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45244)
0929 03:00:49.191857 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.191992 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 03:00:49.193243 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.192860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45246)
0929 03:00:49.193000 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.193108 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 03:00:49.193485 5903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:38861: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
I20260929 03:00:49.261735 5903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } attempt: 2
I20260929 03:00:49.262995 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.262445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45254)
0929 03:00:49.262682 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.262806 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 03:00:49.264077 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.263679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45260)
0929 03:00:49.263839 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.263932 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 03:00:49.264348 5903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:38861: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
I20260929 03:00:49.287832 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } attempt: 5
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650848, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
I20260929 03:00:49.301527 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.288858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.15.18.125:36687)
0929 03:00:49.289044 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.289050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.289089 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:00:49.289168 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.289171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.289234 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.289323 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.290145 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.290838 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.292047 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.292466 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.292573 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.296057 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.296062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.296066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.296100 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.298755 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.299609 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.299613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.299616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.299744 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.300180 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.300184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.300186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.300476 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.300591 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.301115 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.301306 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:00:49.301782 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.288659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:51082)
0929 03:00:49.288865 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.288882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:49.289020 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:49.289376 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:49.289381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:49.289816 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:49.289995 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.290004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.290973 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.290977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.291914 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.291927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.292605 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.292611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.292818 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.293529 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:49.293548 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:49.295836 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:49.298913 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.298918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.298922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.299451 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.299859 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.299864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.299868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.300056 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.300601 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:49.300607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:49.300806 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 03:00:49.300963 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 03:00:49.301343 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 03:00:49.302894 5829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:00:49.304014 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.303586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45276)
0929 03:00:49.303756 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.303849 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:00:49.304894 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.304543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45280)
0929 03:00:49.304669 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.304765 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:00:49.305083 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:38861: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
I20260929 03:00:49.355592 5903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } attempt: 3
I20260929 03:00:49.356297 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } attempt: 1
I20260929 03:00:49.356798 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.356242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45288)
0929 03:00:49.356468 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.356588 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 03:00:49.357421 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.356871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45304)
0929 03:00:49.357120 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.357224 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 03:00:49.357949 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.357438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45316)
0929 03:00:49.357715 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.357813 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
W20260929 03:00:49.358199 5903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:38861: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
I20260929 03:00:49.358420 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.358020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45318)
0929 03:00:49.358213 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.358300 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260929 03:00:49.358597 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:38861: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
I20260929 03:00:49.426823 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } attempt: 2
I20260929 03:00:49.428239 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.427662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45324)
0929 03:00:49.427930 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.428049 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 03:00:49.429538 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.428970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45328)
0929 03:00:49.429276 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.429384 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260929 03:00:49.430440 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:38861: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
W20260929 03:00:49.487056 5910 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:00:49.487398 5910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:00:49.487471 5910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:00:49.487592 5910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:00:49.494839 5910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:00:49.494941 5910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:00:49.494997 5910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:00:49.495048 5910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:00:49.495107 5910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:00:49.501416 5903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } attempt: 4
I20260929 03:00:49.502477 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.502021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45332)
0929 03:00:49.502243 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.502332 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 03:00:49.503415 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.503133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45348)
0929 03:00:49.503245 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.503315 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:00:49.503629 5903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:38861: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
I20260929 03:00:49.505280 5910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.18.84:46563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.18.126:43015,127.15.18.125:36687,127.15.18.124:38861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/krb5kdc/kudu_127.15.18.124.keytab
--principal=kudu/127.15.18.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.18.124:38861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.18.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 02:00:25 UTC on ccycloud.quasar-cofvqt.root.comops.site
build id 4679191
ASAN enabled
I20260929 03:00:49.506541 5910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:00:49.507910 5910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:00:49.519441 5915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:49.519538 5916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:00:49.519479 5918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:00:49.526513 5910 server_base.cc:1034] running on GCE node
I20260929 03:00:49.527384 5910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:00:49.529006 5910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:00:49.530231 5910 hybrid_clock.cc:648] HybridClock initialized: now 1790650849530168 us; error 47 us; skew 500 ppm
I20260929 03:00:49.530563 5910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650849, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:00:49.533716 5910 init.cc:377] Logged in from keytab as kudu/127.15.18.124@KRBTEST.COM (short username kudu)
I20260929 03:00:49.535213 5910 webserver.cc:460] Webserver started at http://127.15.18.124:44965/ using document root <none> and password file <none>
I20260929 03:00:49.535964 5910 fs_manager.cc:359] Metadata directory not provided
I20260929 03:00:49.536068 5910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:00:49.536371 5910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:00:49.536638 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } attempt: 3
I20260929 03:00:49.537598 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.537164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45360)
0929 03:00:49.537343 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.537438 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 03:00:49.538648 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.538271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45364)
0929 03:00:49.538477 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.538548 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 03:00:49.538954 5829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.18.124:38861: Network error: Client connection negotiation failed: client connection to 127.15.18.124:38861: connect: Connection refused (error 111)
I20260929 03:00:49.539712 5910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/data/instance:
uuid: "9445823dc0ba45c6802f4cc16e272c26"
format_stamp: "Formatted at 2026-09-29 03:00:49 on dist-test-slave-q5f0"
server_key: "78460c6a8a8d6e6ee814b271968e752f"
server_key_iv: "0e442efd97f0a2710dc7a35b985e607f"
server_key_version: "encryptionkey@0"
I20260929 03:00:49.540474 5910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/wal/instance:
uuid: "9445823dc0ba45c6802f4cc16e272c26"
format_stamp: "Formatted at 2026-09-29 03:00:49 on dist-test-slave-q5f0"
server_key: "78460c6a8a8d6e6ee814b271968e752f"
server_key_iv: "0e442efd97f0a2710dc7a35b985e607f"
server_key_version: "encryptionkey@0"
I20260929 03:00:49.545485 5910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260929 03:00:49.548941 5925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:00:49.550256 5910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 03:00:49.550412 5910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/data,/tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/wal
uuid: "9445823dc0ba45c6802f4cc16e272c26"
format_stamp: "Formatted at 2026-09-29 03:00:49 on dist-test-slave-q5f0"
server_key: "78460c6a8a8d6e6ee814b271968e752f"
server_key_iv: "0e442efd97f0a2710dc7a35b985e607f"
server_key_version: "encryptionkey@0"
I20260929 03:00:49.550613 5910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:00:49.567936 5910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:00:49.572157 5910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:00:49.572431 5910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:00:49.589782 5910 rpc_server.cc:307] RPC server started. Bound to: 127.15.18.124:38861
I20260929 03:00:49.589787 5976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.18.124:38861 every 8 connection(s)
I20260929 03:00:49.591096 5910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/data/info.pb
I20260929 03:00:49.594688 5977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:00:49.600178 15433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu as pid 5910
I20260929 03:00:49.600364 15433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxb3wIY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790650496826590-15433-0/minicluster-data/master-2/wal/instance
I20260929 03:00:49.600868 15433 external_mini_cluster.cc:1428] Setting key 526c2640a0a74444c23e985bbca45f05
I20260929 03:00:49.601568 5977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650849, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
I20260929 03:00:49.619657 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.604449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.15.18.126:43015)
0929 03:00:49.604665 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.604673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.605301 (+ 628us) server_negotiation.cc:407] Connection header received
0929 03:00:49.606131 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.606136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.606204 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.606322 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.607729 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.608431 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.609913 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.610327 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.610438 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.613964 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.613969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.613972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.614004 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.616653 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.617477 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.617482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.617486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.617611 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.617988 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.617993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.617995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.618259 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.618389 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.619048 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.619269 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 03:00:49.620639 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.602841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.15.18.124:38861)
0929 03:00:49.603613 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.603624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.603663 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:00:49.605814 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.605841 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.606169 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.606352 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.607844 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.609029 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.610234 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.610761 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.610870 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.614554 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.614562 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.614573 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.614604 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.617291 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.617905 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.617911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.617913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.618031 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.618377 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.618381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.618383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.618743 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.619032 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.619316 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.619825 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":535,"thread_start_us":309,"threads_started":1}
I20260929 03:00:49.620673 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.604382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.126:43015 (local address 127.0.0.1:57158)
0929 03:00:49.605116 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.605171 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:49.605825 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:49.606453 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:49.606467 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:49.607113 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:49.607556 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.607582 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.608582 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.608586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.609761 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.609773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.610558 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.610564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.610777 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.611478 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:49.611504 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:49.613774 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:49.616778 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.616790 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.616803 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.617354 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.617713 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.617718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.617722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.617885 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.618386 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:49.618394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:49.618715 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 03:00:49.618952 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 03:00:49.619469 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":310,"threads_started":1}
I20260929 03:00:49.622459 5977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650849, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.126@KRBTEST.COM
I20260929 03:00:49.636044 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.623177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.125:36687 (local address 127.0.0.1:51092)
0929 03:00:49.623341 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.623360 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:49.623478 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:49.623873 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:49.623877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:49.624283 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:49.624473 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.624484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.625456 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.625460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.626449 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.626463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.627114 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.627121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.627313 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.627907 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:49.627924 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:49.630301 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:49.633422 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.633427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.633431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.633935 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.634331 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.634336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.634340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.634516 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.635040 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:49.635045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:49.635212 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 03:00:49.635392 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 03:00:49.635696 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:00:49.636215 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.623371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.15.18.125:36687)
0929 03:00:49.623585 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.623591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.623608 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:00:49.623657 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.623661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.623730 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.623826 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.624616 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.625328 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.626590 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.626987 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.627074 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.630480 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.630485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.630488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.630520 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.633315 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.634047 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.634052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.634056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.634218 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.634636 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.634640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.634642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.634920 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.635031 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.635561 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.635817 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:00:49.637396 5977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:00:49.638936 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.623641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.15.18.126:43015)
0929 03:00:49.623812 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.623817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.623906 (+ 89us) server_negotiation.cc:407] Connection header received
0929 03:00:49.624040 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.624044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.624094 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.624162 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.625313 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.626070 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.627491 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.628040 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.628131 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.632530 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.632535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.632538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.632568 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.635806 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.636719 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.636727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.636730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.636911 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.637357 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.637362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.637366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.637711 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.637982 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.638305 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.638595 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650849, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.124@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.18.125@KRBTEST.COM
I20260929 03:00:49.650550 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.638097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45378)
0929 03:00:49.638307 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.638325 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:49.638428 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:49.638808 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:49.638811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:49.639222 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:49.639365 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.639376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.640436 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.640441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.641372 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.641386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.642301 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.642306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.642474 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.643085 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:49.643101 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:49.645260 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:49.648080 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.648085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.648089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.648526 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.648889 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.648892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.648894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.649011 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.649624 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:49.649627 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:49.649745 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 03:00:49.649871 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 03:00:49.650135 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 03:00:49.650590 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.638148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.15.18.124:38861)
0929 03:00:49.638301 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.638307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.638395 (+ 88us) server_negotiation.cc:407] Connection header received
0929 03:00:49.638533 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.638538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.638594 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.638663 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.639541 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.640309 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.641507 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.642129 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.642268 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.645439 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.645444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.645447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.645477 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.647967 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.648641 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.648645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.648647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.648763 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.649113 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.649118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.649121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.649513 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.649659 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.649974 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.650316 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":48}
I20260929 03:00:49.655414 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.640586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.15.18.125:36687)
0929 03:00:49.640747 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.640752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.640792 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:00:49.640973 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.640978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.641044 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.641123 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.642654 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.643486 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.644949 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.645373 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.645484 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.649824 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.649828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.649831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.649862 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.652889 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.653516 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.653522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.653525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.653709 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.654075 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.654102 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.654106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.654436 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.654676 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.654868 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.655086 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:00:49.656203 5977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26: Bootstrap starting.
I20260929 03:00:49.659665 5977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:49.659982 15433 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:00:49.660874 5977 log.cc:824] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:49.663945 5977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26: No bootstrap required, opened a new log
I20260929 03:00:49.662745 5804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1240553dc9d44fbcb26de6ee8e383f58"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57164
I20260929 03:00:49.674346 5977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } }
I20260929 03:00:49.674758 5977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9445823dc0ba45c6802f4cc16e272c26, State: Initialized, Role: FOLLOWER
I20260929 03:00:49.675310 5977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } }
I20260929 03:00:49.676222 5983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } } }
I20260929 03:00:49.676532 5983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:49.677459 5977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 03:00:49.686156 5994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:00:49.686271 5994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:00:49.684630 5910 master_runner.cc:186] Error getting master registration for 127.15.18.126:43015: OK, instance_id { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" instance_seqno: 1790650848577563 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:00:49.695367 5910 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:00:49.716176 5829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } attempt: 4
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650848, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.126@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
I20260929 03:00:49.733681 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.717018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45392)
0929 03:00:49.717253 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.717278 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:49.717425 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:49.717800 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:49.717806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:49.718335 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:49.718558 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.718572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.720093 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.720099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.721597 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.721614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.722778 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.722785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.723000 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.723737 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:49.723762 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:49.726628 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:49.730647 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.730654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.730659 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.731205 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.731637 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.731643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.731647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.731828 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.732567 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:49.732573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:49.732743 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 03:00:49.732900 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 03:00:49.733209 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 03:00:49.733726 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.717248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.15.18.124:38861)
0929 03:00:49.717456 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.717462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.717486 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:00:49.717553 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.717559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.717640 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.717757 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.718728 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.719889 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.721780 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.722594 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.722749 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.726823 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.726831 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.726835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.726875 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.730479 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.731336 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.731342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.731346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.731495 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.731943 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.731948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.731951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.732433 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.732580 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.733012 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.733251 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 03:00:49.739876 5829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58: Bootstrap starting.
I20260929 03:00:49.744077 5829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:49.745416 5829 log.cc:824] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:49.749068 5829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58: No bootstrap required, opened a new log
I20260929 03:00:49.757019 5829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } }
I20260929 03:00:49.757520 5829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1240553dc9d44fbcb26de6ee8e383f58, State: Initialized, Role: FOLLOWER
I20260929 03:00:49.758102 5829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } }
I20260929 03:00:49.759224 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } } }
I20260929 03:00:49.759661 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:49.760620 5829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:49.760726 5804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:00:49.760958 5804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:49.763792 5804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } }
I20260929 03:00:49.766294 5952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1240553dc9d44fbcb26de6ee8e383f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9445823dc0ba45c6802f4cc16e272c26"
I20260929 03:00:49.766810 5952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:49.770404 5952 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1240553dc9d44fbcb26de6ee8e383f58 in term 1.
I20260929 03:00:49.770884 5804 leader_election.cc:288] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [CANDIDATE]: Term 1 election: Requested vote from peers a4e4b7dbc31a4e25ba566f9d399f0f62 (127.15.18.125:36687), 9445823dc0ba45c6802f4cc16e272c26 (127.15.18.124:38861)
I20260929 03:00:49.771145 5780 leader_election.cc:302] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1240553dc9d44fbcb26de6ee8e383f58, 9445823dc0ba45c6802f4cc16e272c26; no voters:
I20260929 03:00:49.771652 5997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:00:49.784747 5997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [term 1 LEADER]: Becoming Leader. State: Replica: 1240553dc9d44fbcb26de6ee8e383f58, State: Running, Role: LEADER
I20260929 03:00:49.786000 5997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } }
I20260929 03:00:49.800848 6003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1240553dc9d44fbcb26de6ee8e383f58. Latest consensus state: current_term: 1 leader_uuid: "1240553dc9d44fbcb26de6ee8e383f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } } }
I20260929 03:00:49.801370 6003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:49.802840 5903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } attempt: 5
Sep 29 03:00:49 dist-test-slave-q5f0 krb5kdc[5749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650849, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.18.125@KRBTEST.COM for kudu/127.15.18.124@KRBTEST.COM
W20260929 03:00:49.818027 5783 tablet.cc:2369] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:00:49.821046 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.803596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.18.124:38861 (local address 127.0.0.1:45394)
0929 03:00:49.804042 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 03:00:49.804064 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:00:49.804190 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:00:49.804792 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:00:49.804798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:00:49.805318 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:00:49.805542 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.805556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.807056 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.807062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.808481 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:00:49.808499 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.809761 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.809768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:00:49.809996 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.810676 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:00:49.810702 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:00:49.813670 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:00:49.817997 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.818003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.818007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.818575 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.819064 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:00:49.819069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:00:49.819073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:00:49.819250 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:00:49.820002 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:00:49.820007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:00:49.820177 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 03:00:49.820336 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 03:00:49.820646 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309}
I20260929 03:00:49.822070 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.804119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.15.18.124:38861)
0929 03:00:49.804293 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.804299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.804316 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:00:49.804378 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.804395 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.804494 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.804593 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.805739 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.806875 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.808681 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.809394 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.809539 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.813934 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.813941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.813946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.813990 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.817805 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.818740 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.818745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.818749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.818923 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.819384 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.819390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.819394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.819783 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.819931 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.821441 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.821690 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 03:00:49.824419 5931 tablet.cc:2369] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:00:49.828347 5903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62: Bootstrap starting.
I20260929 03:00:49.830521 6013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:00:49.833022 5903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62: Neither blocks nor log segments found. Creating new log.
I20260929 03:00:49.834583 5903 log.cc:824] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62: Log is configured to *not* fsync() on all Append() calls
I20260929 03:00:49.850674 5903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62: No bootstrap required, opened a new log
I20260929 03:00:49.854662 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.813366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.15.18.124:38861)
0929 03:00:49.823220 (+ 9854us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.823225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.832539 (+ 9314us) server_negotiation.cc:407] Connection header received
0929 03:00:49.833034 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.833040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.833138 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.833212 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.837350 (+ 4138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.838431 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.841469 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.842178 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.842273 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.846672 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.846679 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.846684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.846726 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.849939 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.851287 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.851294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.851297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.851457 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.852144 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.852149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.852152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.852513 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.852754 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.853604 (+ 850us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.853995 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9737,"thread_start_us":15233,"threads_started":1}
I20260929 03:00:49.856902 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.801114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.15.18.125:36687)
0929 03:00:49.801312 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.801317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.825211 (+ 23894us) server_negotiation.cc:407] Connection header received
0929 03:00:49.826513 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.826519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.826603 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.826672 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.834302 (+ 7630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.835424 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.838973 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.839604 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.839706 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.849543 (+ 9837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.849551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.849556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.849598 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.853264 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.854113 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.854119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.854122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.854388 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.854871 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.854876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.854879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.855296 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.855597 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.856307 (+ 710us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.856533 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:00:49.870611 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:49.796986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.15.18.126:43015)
0929 03:00:49.797199 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:49.797207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:49.809683 (+ 12476us) server_negotiation.cc:407] Connection header received
0929 03:00:49.810855 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:49.810860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:49.810946 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:49.811021 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:49.821299 (+ 10278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.822613 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.832751 (+ 10138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:49.833629 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:49.833754 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:49.844250 (+ 10496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:00:49.844259 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:00:49.844264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:00:49.844337 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:00:49.848196 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.849217 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.849223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.849227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.863776 (+ 14549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:00:49.868008 (+ 4232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:00:49.868014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:00:49.868018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:00:49.868467 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:00:49.868764 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:49.869781 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0929 03:00:49.870100 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:00:49.871460 5903 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } }
I20260929 03:00:49.876657 6013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:00:49.876947 5903 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e4b7dbc31a4e25ba566f9d399f0f62, State: Initialized, Role: FOLLOWER
I20260929 03:00:49.877624 5903 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } }
I20260929 03:00:49.899408 5762 master_runner.cc:186] Error getting master registration for 127.15.18.125:36687: OK, instance_id { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" instance_seqno: 1790650849075078 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:00:49.912660 5903 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:00:49.912946 5952 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [term 1 FOLLOWER]: Refusing update from remote peer 1240553dc9d44fbcb26de6ee8e383f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:00:49.913089 5878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1240553dc9d44fbcb26de6ee8e383f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62"
I20260929 03:00:49.914083 5997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:00:49.914300 5762 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:00:49.914997 5877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:00:49.920066 5877 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [term 1 FOLLOWER]: Refusing update from remote peer 1240553dc9d44fbcb26de6ee8e383f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:00:49.924110 5878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1240553dc9d44fbcb26de6ee8e383f58 in term 1.
I20260929 03:00:49.925637 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } } }
I20260929 03:00:49.926020 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:49.925545 5983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1240553dc9d44fbcb26de6ee8e383f58. Latest consensus state: current_term: 1 leader_uuid: "1240553dc9d44fbcb26de6ee8e383f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } } }
I20260929 03:00:49.926033 5997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:00:49.926226 5983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:49.933871 6023 mvcc.cc:205] Tried to move back new op lower bound from 7334505881231986688 to 7334505880735821824. Current Snapshot: MvccSnapshot[applied={T|T < 7334505881231986688}]
I20260929 03:00:49.935560 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1240553dc9d44fbcb26de6ee8e383f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } } }
I20260929 03:00:49.935935 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:49.943759 6013 catalog_manager.cc:1347] Generated new cluster ID: 5bab647e95bc4cf2b79cd3c980e41338
I20260929 03:00:49.943892 6013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:00:49.948244 6003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1240553dc9d44fbcb26de6ee8e383f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } } }
I20260929 03:00:49.948506 6003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58 [sys.catalog]: This master's current role is: LEADER
I20260929 03:00:49.949298 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1240553dc9d44fbcb26de6ee8e383f58. Latest consensus state: current_term: 1 leader_uuid: "1240553dc9d44fbcb26de6ee8e383f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } } }
I20260929 03:00:49.949728 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:49.951083 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1240553dc9d44fbcb26de6ee8e383f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } } }
I20260929 03:00:49.951278 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:49.955992 5983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1240553dc9d44fbcb26de6ee8e383f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1240553dc9d44fbcb26de6ee8e383f58" member_type: VOTER last_known_addr { host: "127.15.18.126" port: 43015 } } peers { permanent_uuid: "a4e4b7dbc31a4e25ba566f9d399f0f62" member_type: VOTER last_known_addr { host: "127.15.18.125" port: 36687 } } peers { permanent_uuid: "9445823dc0ba45c6802f4cc16e272c26" member_type: VOTER last_known_addr { host: "127.15.18.124" port: 38861 } } }
I20260929 03:00:49.956261 5983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:00:49.962663 6029 mvcc.cc:205] Tried to move back new op lower bound from 7334505881231986688 to 7334505880735821824. Current Snapshot: MvccSnapshot[applied={T|T < 7334505881231986688 or (T in {7334505881231986688})}]
I20260929 03:00:49.979018 6013 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:00:49.980995 6013 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:00:50.002707 6013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1240553dc9d44fbcb26de6ee8e383f58: Generated new TSK 0
I20260929 03:00:50.003899 6013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:00:50.022855 6042 catalog_manager.cc:1259] Loaded cluster ID: 5bab647e95bc4cf2b79cd3c980e41338
I20260929 03:00:50.022980 6042 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62: loading cluster ID for follower catalog manager: success
I20260929 03:00:50.026929 6042 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62: acquiring CA information for follower catalog manager: success
I20260929 03:00:50.029589 6042 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a4e4b7dbc31a4e25ba566f9d399f0f62: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:00:50.138049 5832 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.125680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.15.18.126:43015)
0929 03:00:50.125886 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.125893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.129619 (+ 3726us) server_negotiation.cc:407] Connection header received
0929 03:00:50.136298 (+ 6679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.136306 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.136422 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.136500 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.137737 (+ 1237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57182: BlockingRecv error: recv got EOF from 127.0.0.1:57182 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 03:00:50.141999 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.126675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.15.18.125:36687)
0929 03:00:50.126868 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.126874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.139008 (+ 12134us) server_negotiation.cc:407] Connection header received
0929 03:00:50.140420 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.140426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.140513 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.140589 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.141745 (+ 1156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51108: BlockingRecv error: recv got EOF from 127.0.0.1:51108 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 03:00:50.144574 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.139656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.15.18.124:38861)
0929 03:00:50.139832 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.139838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.142595 (+ 2757us) server_negotiation.cc:407] Connection header received
0929 03:00:50.143241 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.143247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.143329 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.143403 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.144304 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45410: BlockingRecv error: recv got EOF from 127.0.0.1:45410 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 03:00:50.188982 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.173059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.15.18.125:36687)
0929 03:00:50.173254 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.173261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.180201 (+ 6940us) server_negotiation.cc:407] Connection header received
0929 03:00:50.180789 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.180796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.181029 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.181175 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.181904 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.183114 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.185716 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.186398 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.186503 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.187328 (+ 825us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.188109 (+ 781us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.188256 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.188499 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.188597 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 03:00:50.232228 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.222315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.15.18.124:38861)
0929 03:00:50.222540 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.222547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.230022 (+ 7475us) server_negotiation.cc:407] Connection header received
0929 03:00:50.230605 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.230612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.230751 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.230828 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.231972 (+ 1144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45416: BlockingRecv error: recv got EOF from 127.0.0.1:45416 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 03:00:50.275007 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.260714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.15.18.125:36687)
0929 03:00:50.260942 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.260949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.268038 (+ 7089us) server_negotiation.cc:407] Connection header received
0929 03:00:50.268358 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.268364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.268465 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.268535 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.269313 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.270548 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.272716 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.273447 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.273595 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.274128 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.274321 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.274422 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.274582 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.274677 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 03:00:50.284833 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.283162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.15.18.124:38861)
0929 03:00:50.283368 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.283374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.283395 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:00:50.283472 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.283478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.283622 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.283731 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.284603 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45426: BlockingRecv error: recv got EOF from 127.0.0.1:45426 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 03:00:50.302162 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.290503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.15.18.125:36687)
0929 03:00:50.290673 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.290680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.296147 (+ 5467us) server_negotiation.cc:407] Connection header received
0929 03:00:50.296349 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.296355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.296451 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.296568 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.297056 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.298173 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.300092 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.300728 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.300857 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.301310 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.301479 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.301575 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.301714 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.301807 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 03:00:50.311826 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.309534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.15.18.124:38861)
0929 03:00:50.309716 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.309722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.310249 (+ 527us) server_negotiation.cc:407] Connection header received
0929 03:00:50.310438 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.310444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.310545 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.310659 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.311574 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45428: BlockingRecv error: recv got EOF from 127.0.0.1:45428 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 03:00:50.330119 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.322399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.15.18.125:36687)
0929 03:00:50.322641 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.322647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.323183 (+ 536us) server_negotiation.cc:407] Connection header received
0929 03:00:50.323557 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.323563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.323664 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.323789 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.324277 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.325714 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.327860 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.328505 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.328642 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.329246 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.329396 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.329495 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.329760 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.329851 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260929 03:00:50.353837 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.349293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.15.18.124:38861)
0929 03:00:50.349593 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.349599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.352149 (+ 2550us) server_negotiation.cc:407] Connection header received
0929 03:00:50.352385 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.352392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.352506 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.352650 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.353577 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45442: BlockingRecv error: recv got EOF from 127.0.0.1:45442 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
I20260929 03:00:50.372838 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.365324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.15.18.125:36687)
0929 03:00:50.365508 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.365514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.366349 (+ 835us) server_negotiation.cc:407] Connection header received
0929 03:00:50.366557 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.366565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.366667 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.366798 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.367267 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.368495 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.370464 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.371147 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.371321 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.371857 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.372044 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.372159 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.372345 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.372442 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 03:00:50.383352 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.380842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.15.18.124:38861)
0929 03:00:50.381035 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.381042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.381635 (+ 593us) server_negotiation.cc:407] Connection header received
0929 03:00:50.381864 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.381870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.381975 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.382099 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.383089 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45458: BlockingRecv error: recv got EOF from 127.0.0.1:45458 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 03:00:50.411088 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.403700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.15.18.125:36687)
0929 03:00:50.403888 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.403895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.404474 (+ 579us) server_negotiation.cc:407] Connection header received
0929 03:00:50.404702 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.404708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.404815 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.404932 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.405462 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.406682 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.408787 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.409516 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.409658 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.410266 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.410424 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.410521 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.410689 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.410787 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 03:00:50.421141 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.418634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.15.18.124:38861)
0929 03:00:50.418825 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.418831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.419373 (+ 542us) server_negotiation.cc:407] Connection header received
0929 03:00:50.419610 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.419618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.419739 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.419859 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.420835 (+ 976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45464: BlockingRecv error: recv got EOF from 127.0.0.1:45464 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:00:50.463129 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.457202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.15.18.125:36687)
0929 03:00:50.457426 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.457432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.457730 (+ 298us) server_negotiation.cc:407] Connection header received
0929 03:00:50.457897 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.457904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.458013 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.458134 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.458514 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.459731 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.461125 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.461873 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.462007 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.462359 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.462499 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.462592 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.462689 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.462777 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 03:00:50.470670 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.468732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.15.18.124:38861)
0929 03:00:50.468931 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.468936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.469338 (+ 402us) server_negotiation.cc:407] Connection header received
0929 03:00:50.469467 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.469471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.469566 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.469654 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.470414 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45474: BlockingRecv error: recv got EOF from 127.0.0.1:45474 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 03:00:50.543449 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.538504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.15.18.125:36687)
0929 03:00:50.538761 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.538767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.539042 (+ 275us) server_negotiation.cc:407] Connection header received
0929 03:00:50.539215 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.539219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.539292 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.539371 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.539792 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.540510 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.541872 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.542314 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.542414 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.542788 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.542892 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.542962 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.543066 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.543133 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 03:00:50.550488 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.548523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.15.18.124:38861)
0929 03:00:50.548798 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.548805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.549135 (+ 330us) server_negotiation.cc:407] Connection header received
0929 03:00:50.549338 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.549344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.549435 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.549543 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.550267 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45486: BlockingRecv error: recv got EOF from 127.0.0.1:45486 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 03:00:50.686914 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.681924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51198 (local address 127.15.18.125:36687)
0929 03:00:50.682124 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.682131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.682468 (+ 337us) server_negotiation.cc:407] Connection header received
0929 03:00:50.682661 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.682666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.682735 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.682814 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.683246 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.683976 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.685307 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.685703 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.685813 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.686213 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.686332 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.686406 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.686549 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.686634 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:00:50.688565 5994 catalog_manager.cc:1259] Loaded cluster ID: 5bab647e95bc4cf2b79cd3c980e41338
I20260929 03:00:50.688660 5994 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26: loading cluster ID for follower catalog manager: success
I20260929 03:00:50.691751 5994 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26: acquiring CA information for follower catalog manager: success
W20260929 03:00:50.694069 5980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:00:50.692416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.15.18.124:38861)
0929 03:00:50.692655 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.692661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.692877 (+ 216us) server_negotiation.cc:407] Connection header received
0929 03:00:50.693024 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.693029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.693116 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.693235 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:00:50.693859 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45498: BlockingRecv error: recv got EOF from 127.0.0.1:45498 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 03:00:50.694113 5994 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9445823dc0ba45c6802f4cc16e272c26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:00:50.959137 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.953954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.15.18.125:36687)
0929 03:00:50.954160 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.954166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.954389 (+ 223us) server_negotiation.cc:407] Connection header received
0929 03:00:50.954637 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.954641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.954711 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.954790 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.955322 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.956079 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.957465 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.957916 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.958028 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.958458 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.958566 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.958642 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.958790 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.958873 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:00:50.970775 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.965089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.15.18.124:38861)
0929 03:00:50.965259 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.965264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.965570 (+ 306us) server_negotiation.cc:407] Connection header received
0929 03:00:50.965747 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.965752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.965930 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.966031 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.966459 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.967186 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.968639 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.969041 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.969137 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.969620 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.970255 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.970367 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.970464 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.970537 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:00:50.983415 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.977925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.15.18.124:38861)
0929 03:00:50.978104 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.978111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.978658 (+ 547us) server_negotiation.cc:407] Connection header received
0929 03:00:50.978849 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.978854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.978940 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.979018 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.979584 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.980383 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.981801 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.982232 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.982340 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.982758 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.982860 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.982935 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.983063 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.983140 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:00:50.984913 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.976420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51214 (local address 127.15.18.125:36687)
0929 03:00:50.976605 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.976612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.977909 (+ 1297us) server_negotiation.cc:407] Connection header received
0929 03:00:50.978176 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.978184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.978278 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.978342 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.979145 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.980341 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.982708 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.983389 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.983485 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.984186 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.984322 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.984417 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.984546 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.984627 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 03:00:50.993225 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:00:50.976425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.15.18.126:43015)
0929 03:00:50.987901 (+ 11476us) server_negotiation.cc:206] Beginning negotiation
0929 03:00:50.987906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:00:50.987923 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:00:50.988016 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:00:50.988020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:00:50.988170 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:00:50.988308 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:00:50.988749 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.989534 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.991060 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:00:50.991614 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:00:50.991710 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:00:50.992245 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:00:50.992608 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:00:50.992696 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:00:50.992873 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:00:50.992943 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11337,"thread_start_us":11250,"threads_started":1}
I20260929 03:00:51.006389 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5762
I20260929 03:00:51.007493 5824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:51.122095 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 5762
W20260929 03:00:51.138659 5852 connection.cc:535] server connection from 127.0.0.1:51082 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:00:51.139225 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5836
I20260929 03:00:51.140298 5898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:51.248611 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 5836
I20260929 03:00:51.265208 15433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu pid 5910
I20260929 03:00:51.266323 5972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 03:00:51.373174 15433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxb3wIY/build/asan/bin/kudu with pid 5910
2026-09-29T03:00:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4487 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8670 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (354509 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST